diff --git a/docs/adding-a-metric.md b/docs/adding-a-metric.md index f796168a..d6ff1a9c 100644 --- a/docs/adding-a-metric.md +++ b/docs/adding-a-metric.md @@ -31,6 +31,13 @@ For a metric that flows end to end (analyzer → API → treemap, Hot Spots, exp - Confirm the metric appears in the **Color** and **Rank by** pickers (Rank by only when it isn't neutral), ranks Hot Spots with its unit and N/A group, and shows in the tooltip and code viewer for the languages that measure it. - Avoid `// @ts-ignore` / `// @ts-expect-error`; instead, update types so the pipeline is type-safe from API response through to the UI. +## How ranking treats a metric + +- Ranking works on one cohort at a time: files or functions, never both mixed. +- A measured zero stays in the denominator. N/A drops out of the ranking and is listed apart. +- The client (`client/src/utils/hotSpotRanking.ts`) and the report (`server/app/services/reporting.py`) must give the same order. `tests/test_ranking_parity.py` checks this. `client/src/utils/hotSpotRanking.test.ts` covers cohorts, zeros, N/A and neutral metrics. +- Treemap areas: a scope with children gets a `(body)` child for its remaining LOC (`client/src/viz/treemap/utils/tree.ts`), so leaf areas add up to the scope. + ## What the catalog keeps, and why Check a new metric against these decisions before you add it. diff --git a/docs/initiative-history.json b/docs/initiative-history.json index c1052fe8..fcfbc271 100644 --- a/docs/initiative-history.json +++ b/docs/initiative-history.json @@ -106,5 +106,21 @@ "Detect components that return .map(...) of JSX (explorEDA DslCode.tsx).", "Run the next pilot on a TS repo without tsc or eslint and a non-author repo, with a second reviewer." ] + }, + { + "title": "Trustworthy metrics and ranking", + "slug": "metric-trust", + "outcome": "completed", + "retired_at": "2026-11-12", + "summary": "The metric-trust plan was carried out inside the metric system overhaul rather than on its own. Measured zero versus N/A, executable-scope counts, witnesses and one consistent ranking story all shipped in #73 and #75. Reconciled against origin/main 0bfb05ba.", + "reason": "Replaced by the metric system overhaul: its milestones shipped as #73 (zero vs N/A, witnesses, cohorts) and #75 (one catalog, report and client parity).", + "documentation": [ + "docs/adding-a-metric.md", + "docs/research/metrics-2026-10-15/README.md" + ], + "revision": "0bfb05ba", + "follow_ups": [ + "Dropped: a declared display-area ownership rule with area-union fixtures (m2). Nothing tracks it." + ] } ] diff --git a/docs/intent/metric-trust/implementation-plan.md b/docs/intent/metric-trust/implementation-plan.md deleted file mode 100644 index 6cedbe23..00000000 --- a/docs/intent/metric-trust/implementation-plan.md +++ /dev/null @@ -1,57 +0,0 @@ -# Trustworthy metrics and ranking — implementation plan - -## Plan at a glance - -Prove one trustworthy scan/report before changing the full catalog. Then correct units and ownership. Finally align ranking and explorer behavior. Each milestone leaves a usable source-only tool. No new external data provider is required. - -## Implementation strategy - -Use `analysis_types.py` and `models.py` for executable kind and support. Keep availability compact: declared analyzer capabilities plus explicit failed or partial observations. Avoid per-value wrappers unless the first fixture proves they are necessary. - -Use `analysis.py` for ownership and aggregation, `reporting.py` for report selection, and `metricsStore.tsx` for labels and eligibility. Share fixture inputs and expected decisions across languages; do not duplicate a full registry generation system. - -Fast local loop: `cd server && uv run pytest tests/test_complexity_semantics.py tests/test_synthetic_scopes.py tests/test_reporting.py`. Client checks: `cd client && pnpm test`. Inject unsupported, skipped, failed, and partial states through fixture analyzers. These commands require installed local dependencies but no service or network once prepared. - -Rollback uses coherent commits. No deployed service or database target is involved. Browser proof uses a local fixture directory with no private data, API on port 8000 and Vite on 5173. - -## Milestone 1: A scan distinguishes executable evidence from unavailable data - -Change `analysis_types.py`, analyzer scope conversion in `analysis.py`, and the `Node`/`Metrics` contract. Preserve origin and source range; distinguish executable functions from classes, objects, imports and bodies. Define a deterministic per-snapshot scope key from path, kind, range and sibling ordinal. Do not promise identity across refactors. - -Carry supported zero and unavailable states into a report and one explorer detail row. Use a TS fixture, a Python fixture, and a skipped file. Validate nested methods and module logic explicitly. Update `docs/CODE_MAP.md` when main files change. - -### Desired end state - -- The API, report and explorer distinguish measured zero from N/A. -- Maximum complexity points to an executable source location. -- Focused pytest and one local browser inspection prove the same fixture values. - -## Milestone 2: Measures conserve their declared units - -Reproduce equal-span child, overlapping imports, nested JSX, blank/comment spans, and two Markdown data URLs on one line. Extend `test_function_body_loc.py` and `test_treemap_scopes.py` only for failures that matter. - -Declare one display-area ownership rule in `analysis.py` and `client/src/viz/treemap/utils/tree.ts`. Keep source ranges independent of area. Correct real-function count/mean, own nesting and arity. Publish exact TS/Python/Lizard support. Either add a useful TS rollup or display N/A at that scope. Correct misleading labels and README complexity wording. - -### Desired end state - -- Rendered leaf areas sum to the declared parent measure on the proof cases. -- Function summaries use the same eligible population as their counts. -- Every current metric has a tested value or explicit unsupported scope. - -## Milestone 3: Ranking and selection tell one consistent story - -Change `reporting.py:rank_nodes`, CLI metric validation in `run.py`, and `Explorer.tsx`. Specify file/function cohorts, eligibility, direction, max normalization, ties and visible path priority. Default to single-metric ranking. If combinations remain, include supported zero inputs in a fixed declared denominator. Compare only equal availability sets; show incomplete evidence separately. - -Use a small shared cohort to prove UI/report order parity and monotonicity with fixed peers. Remove automatic comment/TODO/class/type/import-count penalties. Keep those values searchable in neutral detail. Reproduce current-selection and filter issues before changing `MetricsSidebar.tsx`, `dataProcessing.ts`, and treemap scales. State whether normalization uses the visible cohort; show that choice in the view and report. - -### Desired end state - -- Increasing a supported risk input cannot lower its score with fixed peers. -- A filtered fixture has matching list, map detail and headless evidence. -- Local browser use verifies selection, N/A, source navigation and unit labels. `pnpm build`, client tests and relevant backend tests pass. - -## Below the cut line - -- New metric families, general type checking, compatibility layers and registry generators. -- Persistent review states, longitudinal identity, trend services and a visual redesign. -- Automatic metric deletion or default selection based solely on correlation. diff --git a/docs/intent/metric-trust/initiative-map.json b/docs/intent/metric-trust/initiative-map.json deleted file mode 100644 index 9f13e9cf..00000000 --- a/docs/intent/metric-trust/initiative-map.json +++ /dev/null @@ -1,265 +0,0 @@ -{ - "schema_version": "0.5.0", - "initiative": { - "title": "Trustworthy metrics and ranking", - "slug": "metric-trust", - "scale": "capability", - "current_phase": "plan", - "last_updated": "2026-10-15", - "summary": "Correct measurement meaning, availability, ownership and comparison before adding metric families.", - "status": "planned", - "next_action": "Review the proposed scope, then execute the measured-zero versus unavailable scan proof." - }, - "artifacts": { - "intent_brief": { - "path": "intent-brief.md", - "status": "current", - "updated_at": "2026-10-15" - }, - "brainstorm": { - "path": "brainstorm.md", - "status": "not-requested", - "updated_at": null - }, - "shape_brief": { - "path": "shape-brief.md", - "status": "current", - "updated_at": "2026-10-15" - }, - "implementation_plan": { - "path": "implementation-plan.md", - "status": "current", - "updated_at": "2026-10-15" - } - }, - "sources": [ - { - "id": "user", - "type": "conversation", - "location": "Current Codex chat", - "anchor": "2026-10-15 original unique-information request and full-process approval", - "authority": "user" - }, - { - "id": "research", - "type": "research", - "location": "docs/research/metrics-2026-10-15/originals/recommendations.md", - "anchor": "R01, R02, R03, R04, R05, R06, R07, R08, R18", - "authority": "external proposals checked against baseline; not product intent" - }, - { - "id": "repo", - "type": "repository", - "location": "server/app/models.py; server/app/services/analysis_types.py; server/app/services/analysis.py; server/app/services/reporting.py; client/src/components/Explorer.tsx; client/src/utils/metricsStore.tsx", - "anchor": "83dcbf40879ca06ab07cdfcf365b5c0ccb7608a4", - "authority": "current source" - } - ], - "intent": { - "claims": [ - { - "id": "C1", - "statement": "Provide unique actionable information per metric.", - "kind": "outcome", - "commitment": "explicit", - "importance": "core", - "status": "active", - "time_horizon": "first release", - "sources": [ - "user" - ], - "confidence": "high" - }, - { - "id": "C2", - "statement": "Avoid several highly correlated metrics that add little information.", - "kind": "anti-goal", - "commitment": "explicit", - "importance": "core", - "status": "active", - "time_horizon": "first release", - "sources": [ - "user" - ], - "confidence": "high" - }, - { - "id": "C3", - "statement": "Review current metrics and the field including CRAP before proposing initiatives.", - "kind": "sequence", - "commitment": "explicit", - "importance": "core", - "status": "active", - "time_horizon": "proof", - "sources": [ - "user" - ], - "confidence": "high" - }, - { - "id": "C4", - "statement": "Research and planning are authorized; application implementation remains planned.", - "kind": "constraint", - "commitment": "explicit", - "importance": "core", - "status": "active", - "time_horizon": "proof", - "sources": [ - "user" - ], - "confidence": "high" - }, - { - "id": "C5", - "statement": "Correct measurement meaning, availability, ownership and comparison before adding metric families.", - "kind": "implementation hypothesis", - "commitment": "derived", - "importance": "supporting", - "status": "active", - "time_horizon": "proof", - "sources": [ - "research", - "repo" - ], - "confidence": "medium" - } - ], - "themes": [], - "conflicts": [], - "human_questions": [] - }, - "brainstorm": { - "status": "not-requested", - "requested_count": 7, - "options": [], - "selected_option": null - }, - "shape": { - "status": "current", - "appetite": "Small isolated local proofs; no enterprise infrastructure or compatibility layers.", - "problem": "Correct measurement meaning, availability, ownership and comparison before adding metric families.", - "recommended_shape": "Trustworthy metrics and ranking", - "success_moves": [ - "Start with one real scan/report instead of redesigning all analyzers.", - "Use capability declarations and explicit exceptional states before per-value wrappers.", - "Use one current contract and direct ranking implementations with shared fixtures.", - "Reproduce area and selection failures before changing their code." - ], - "first_proof": { - "question": "Can one scan preserve executable ownership and distinguish zero from unavailable end to end?", - "artifact": "TS/Python scan fixture, report and explorer detail.", - "decision_rule": "Proceed only if all expected populations, states and maximum source locations agree." - }, - "rabbit_holes": [], - "open_decisions": [] - }, - "implementation": { - "status": "current", - "repository_snapshot": { - "root": "/Users/byronwall/.codex/worktrees/46c0/srcly", - "revision": "83dcbf40879ca06ab07cdfcf365b5c0ccb7608a4", - "worktree_status": "Research/planning documents only; no application implementation.", - "subsystems": [ - "server/app/models.py", - "server/app/services/analysis_types.py", - "server/app/services/analysis.py", - "server/app/services/reporting.py", - "client/src/components/Explorer.tsx", - "client/src/utils/metricsStore.tsx" - ], - "test_commands": [ - "cd server && uv run pytest tests/test_complexity_semantics.py tests/test_synthetic_scopes.py tests/test_reporting.py", - "cd client && pnpm test", - "cd client && pnpm build" - ], - "deployment_surfaces": [] - }, - "dependencies": [ - { - "id": "analyzers", - "name": "Existing analyzers and scope/area attribution", - "material": true, - "application_seam": "analysis_types.py scope identity and analysis.py node construction", - "first_proof_role": { - "mode": "real", - "rationale": "The first outcome depends on actual analyzer output; hand-built nodes alone cannot establish propagation." - }, - "failure_modes": [ - "unsupported syntax", - "parser partial result", - "skipped file", - "overlapping spans" - ], - "proof_rungs": [ - "local-fixture", - "contract-test", - "real-local-output" - ], - "local_proof": "Small checked-in source fixtures with expected populations and exceptional analyzer outcomes.", - "live_proof": "No external provider; run installed analyzers in the local scan and browser fixture.", - "rollout": "Optional input or coherent commit; omit/revert without parallel API support." - } - ], - "fast_local_loop": { - "command": "cd server && uv run pytest tests/test_complexity_semantics.py tests/test_synthetic_scopes.py tests/test_reporting.py", - "network_required": false, - "shared_infrastructure_required": false, - "failure_injection": [ - "unsupported syntax", - "parser partial result", - "skipped file", - "overlapping spans" - ], - "status": "Proposed commands for implementation; new scripts/tests do not exist yet." - }, - "milestones": [ - { - "id": "m1", - "title": "A scan distinguishes executable evidence from unavailable data", - "outcome": "A scan/report/explorer distinguishes real executable scopes, measured zero and unsupported analysis.", - "dependencies_introduced": [], - "proof": "Focused pytest and local browser check on TS/Python plus skipped-file fixture.", - "valid_intermediate_state": "A scan/report/explorer distinguishes real executable scopes, measured zero and unsupported analysis. Existing source-only scanning remains usable.", - "rollback": "Revert the scoped milestone commit or omit optional evidence; no stored-data migration.", - "status": "planned", - "evidence": [] - }, - { - "id": "m2", - "title": "Measures conserve their declared units", - "outcome": "Source units, display area, counts and per-function summaries agree with hand-declared cases.", - "dependencies_introduced": [], - "proof": "Area-union fixtures, real-function denominator cases and all metric support assertions.", - "valid_intermediate_state": "Source units, display area, counts and per-function summaries agree with hand-declared cases. Existing source-only scanning remains usable.", - "rollback": "Revert the scoped milestone commit or omit optional evidence; no stored-data migration.", - "status": "planned", - "evidence": [] - }, - { - "id": "m3", - "title": "Ranking and selection tell one consistent story", - "outcome": "UI and report compare the same cohort with monotone scores and current selected evidence.", - "dependencies_introduced": [], - "proof": "Shared cohort UI/report parity, fixed-peer monotonicity, selected-scope/filter browser checks.", - "valid_intermediate_state": "UI and report compare the same cohort with monotone scores and current selected evidence. Existing source-only scanning remains usable.", - "rollback": "Revert the scoped milestone commit or omit optional evidence; no stored-data migration.", - "status": "planned", - "evidence": [] - } - ], - "cut_line": [ - "New metric families, general type checking, compatibility layers and registry generators.", - "Persistent review states, longitudinal identity, trend services and a visual redesign.", - "Automatic metric deletion or default selection based solely on correlation." - ], - "open_decisions": [], - "initiative_dependencies": [] - }, - "history": [ - { - "date": "2026-10-15", - "change": "Created planned initiative from collected Pro research and independent source checks. Application proofs remain unrun." - } - ], - "closure": null -} diff --git a/docs/intent/metric-trust/intent-brief.md b/docs/intent/metric-trust/intent-brief.md deleted file mode 100644 index 4fde014b..00000000 --- a/docs/intent/metric-trust/intent-brief.md +++ /dev/null @@ -1,34 +0,0 @@ -# Trustworthy metrics and ranking - -## My read - -Byron wants Srcly to reveal useful code quality information with little duplicated evidence. That starts with measurements whose meaning remains clear across files, functions, folders, reports, and the explorer. More metrics cannot help when a missing value looks healthy or a displayed function is actually a markup container. - -This initiative gives the existing measurements a dependable meaning. A reviewer should see what was measured, where it came from, and which source unit produced a maximum. A failed or unsupported analysis must remain visible. Counts that describe code should not imply a defect without evidence. - -The main user is a developer exploring a local repository. The headless report must serve the same review decision as the browser. It should not assign a different priority merely because its scoring path uses another denominator. These improvements should keep Srcly useful before any history, coverage, clone, or semantic analyzer integration exists. - -## What matters most - -- Separate real executable functions from containers and display fragments. -- Distinguish measured zero from unavailable or partial analysis. -- Use explicit units, comparison populations, and source locations. -- Stop repeated evidence from receiving extra priority through combined scores. - -## Boundaries - -Fix the current contract directly. Assume one client and server version. Do not introduce compatibility adapters, a general schema platform, or a replacement analyzer engine. - -Only add extraction or rollups that support a useful decision. Some TS counters can remain file-only diagnostics. Reproduce suspected source-area and navigation errors before treating them as confirmed product defects. - -Do not add a universal health score. Preserve useful counters as neutral detail when their quality interpretation is weak. Permanent catalog reduction belongs to the portfolio evaluation. - -## What seems settled - -The baseline has separate maximum and average complexity fields. Keep that distinction and correct stale documentation. Ten TS counters do not reach folders. Report ranking omits zero-valued metrics from its weight denominator. These source findings justify focused correctness work. - -Display area, physical lines, nonblank lines, and code lines can serve different purposes. Their labels and ownership rules must make those differences explicit. - -## Next step after confirmation - -Start with one scan fixture that contains real functions, display scopes, a genuine zero, and unavailable analysis. Carry it through API, report, and explorer before broadening coverage. The [shape](shape-brief.md) and [plan](implementation-plan.md) describe the proposed proof sequence. All implementation remains planned. diff --git a/docs/intent/metric-trust/shape-brief.md b/docs/intent/metric-trust/shape-brief.md deleted file mode 100644 index f6bdc4c5..00000000 --- a/docs/intent/metric-trust/shape-brief.md +++ /dev/null @@ -1,51 +0,0 @@ -# Trustworthy metrics and ranking — shape - -**Outcome:** Each displayed value has a clear meaning and leads to the source evidence that produced it. - -**Primary flow:** Scan → choose files or functions → inspect a metric → open its source and support details. - -## Feature scope - -```text -Trustworthy explorer and reports -├── PLANNED ADDITIONS -│ ├── Scope and support -│ │ ├── Real functions distinguished from display containers -│ │ └── Measured zero, unsupported, skipped, partial and failed states -│ ├── Honest measurements -│ │ ├── Explicit line units and conserved display area -│ │ └── Maximum with source location; consistent counts and means -│ └── Explainable comparison -│ ├── Declared file/function cohort and metric eligibility -│ ├── Matching report/list order and visible score contributions -│ └── Neutral diagnostics, current selection and matching filters -└── EVALUATE BEFORE COMMITTING - └── Scope-level TS counters only when they support useful inspection -``` - -## Behavior - -| Situation | Expected result | -| --- | --- | -| A metric does not apply | Show N/A and exclude it from numeric comparisons. | -| A supported count equals zero | Show zero; keep the declared denominator. | -| A folder shows maximum complexity | Open the responsible executable function. | -| A user filters or changes scope | Show the comparison population and current selected source. | - -## Decisions and boundaries - -**Appetite:** Correct existing semantics and presentation before adding another metric family. - -**Key decision:** Share a written ranking specification and fixtures across Python and TypeScript. Use direct implementations rather than a cross-language scoring framework. Default to one explicit metric; an optional combination must expose its inputs. - -**Boundary:** Keep the existing layout. Use one current API contract. Revert a failed slice as a commit; do not maintain parallel versions. Persistent triage and trends remain outside this initiative. - -**First proof:** One small real scan and its report. - -**Try:** Scan a TS/Python fixture with nested functions, display scopes, measured zero, and skipped analysis. - -**Observe:** Check the executable population, availability labels, source locations, and report values by hand. - -**Decide:** Continue only if zero and unavailable remain distinct end to end and each maximum identifies the correct function. - -See the [implementation plan](implementation-plan.md). diff --git a/docs/research/metrics-2026-10-15/README.md b/docs/research/metrics-2026-10-15/README.md index 1b9c762d..c7587bc7 100644 --- a/docs/research/metrics-2026-10-15/README.md +++ b/docs/research/metrics-2026-10-15/README.md @@ -8,7 +8,7 @@ The research covers 32 API fields, 27 report definitions, 22 selectable metrics, | Initiative | Result | First proof | | --- | --- | --- | -| [Metric trust](../../intent/metric-trust/shape-brief.md) | Correct scope, units, support, rollups, and ranking | A measured zero differs from unavailable data in one real scan and report | +| [Metric trust](https://github.com/byronwall/srcly/blob/0bfb05ba/docs/intent/metric-trust/shape-brief.md) | Correct scope, units, support, rollups, and ranking | A measured zero differs from unavailable data in one real scan and report | | [Portfolio evaluation](../../intent/metric-portfolio/shape-brief.md) | Evidence for a compact default and useful diagnostics | Three pinned repositories reveal retained and rejected decision cases | | [Git activity](../../intent/git-activity/shape-brief.md) | Explain why difficult code deserves attention now | Verified file histories change a justified priority decision | | [Imported quality evidence](../../intent/imported-quality-evidence/shape-brief.md) | Inspect cycles, clones, diagnostics, and test gaps | An existing graph edge leads to its original source declaration |