From ccda11d907eead865ed703988d7811883f25d0df Mon Sep 17 00:00:00 2001 From: Byron Wall Date: Thu, 12 Nov 2026 21:55:11 -0500 Subject: [PATCH] docs: retire metric-portfolio The metric system overhaul replaced this plan: #74, #75 and #79 shipped its milestones. Move the durable facts into the lasting docs, add the closure record to docs/initiative-history.json, and remove the planning folder. Co-Authored-By: Claude Opus 5.5 --- docs/adding-a-metric.md | 1 + docs/initiative-history.json | 16 ++ .../metric-portfolio/implementation-plan.md | 58 ---- .../metric-portfolio/initiative-map.json | 267 ------------------ docs/intent/metric-portfolio/intent-brief.md | 32 --- docs/intent/metric-portfolio/shape-brief.md | 42 --- docs/research/metrics-2026-10-15/README.md | 2 +- 7 files changed, 18 insertions(+), 400 deletions(-) delete mode 100644 docs/intent/metric-portfolio/implementation-plan.md delete mode 100644 docs/intent/metric-portfolio/initiative-map.json delete mode 100644 docs/intent/metric-portfolio/intent-brief.md delete mode 100644 docs/intent/metric-portfolio/shape-brief.md diff --git a/docs/adding-a-metric.md b/docs/adding-a-metric.md index d6ff1a9c..0222876e 100644 --- a/docs/adding-a-metric.md +++ b/docs/adding-a-metric.md @@ -49,4 +49,5 @@ Check a new metric against these decisions before you add it. - Findings come from the project's own `tsc` and `eslint` when present. Otherwise srcly runs `ruff` and its own TypeScript rules. Each finding names its tool. - Named 2x2 presets use absolute cut-offs, not cut-offs relative to the repo. - Below the cut line: clones, coverage, CRAP, temporal coupling and trends. +- Keep or cut a metric on decision cases: read each top-ranked case in source and record a verdict. Correlation alone does not decide. `evaluate_metric_portfolio.py` reports support, N/A and zero rates, Spearman correlation and top-10 overlap per language and scope as input to that review. - Tune defaults with evidence. The October 2026 pilot adjudicated 729 top-10 cases on four repositories, then set the defaults (for example, `comment_noise` flags 1.0 comment line per code line or more). Its method, results and open follow-ups are in `docs/research/metric-pilot-2026-10/README.md`. Rerun a pilot with `server/scripts/evaluate_metric_portfolio.py`. diff --git a/docs/initiative-history.json b/docs/initiative-history.json index fcfbc271..7b4a42c1 100644 --- a/docs/initiative-history.json +++ b/docs/initiative-history.json @@ -122,5 +122,21 @@ "follow_ups": [ "Dropped: a declared display-area ownership rule with area-union fixtures (m2). Nothing tracks it." ] + }, + { + "title": "A metric portfolio that earns its attention", + "slug": "metric-portfolio", + "outcome": "completed", + "retired_at": "2026-11-12", + "summary": "The portfolio evaluation and catalog choice shipped inside the metric system overhaul: an evaluation script, a four-repo pilot with source-backed verdicts, and the cuts and weights it justified. Reconciled against origin/main 0bfb05ba.", + "reason": "Replaced by the metric system overhaul: #74 and #75 applied the catalog choice and #79 ran the multi-repo evaluation.", + "documentation": [ + "docs/adding-a-metric.md", + "docs/research/metric-pilot-2026-10/README.md" + ], + "revision": "0bfb05ba", + "follow_ups": [ + "The pilot README lists its own follow-ups, including a second reviewer and a non-author repository." + ] } ] diff --git a/docs/intent/metric-portfolio/implementation-plan.md b/docs/intent/metric-portfolio/implementation-plan.md deleted file mode 100644 index 9f56dccb..00000000 --- a/docs/intent/metric-portfolio/implementation-plan.md +++ /dev/null @@ -1,58 +0,0 @@ -# A useful metric portfolio — implementation plan - -## Plan at a glance - -Prove the comparison method on known cases. Then run a three-repository pilot. Only then choose defaults or expand the sample. This prevents broken measurement support from producing persuasive but false statistical results. - -## Implementation strategy - -Depend on metric trust milestone 2 for reliable scope, units and availability. Reuse `reporting.py` exports. Add one standalone Python evaluation script, proposed as `server/scripts/evaluate_metric_portfolio.py`, and small input artifacts. Do not add a service or database. - -The first proof uses deterministic saved observations. Proposed local command after implementation: `cd server && uv run python scripts/evaluate_metric_portfolio.py --manifest ../tmp/portfolio/manifest.json --out ../tmp/portfolio/results`. Record dependencies and run instructions with the script. Use existing Python tools before adding a statistics package. - -Use within-stratum Spearman and top-k overlap at k=10 and 25. Report n, ties, support and zero rates. For size adjustment compare log-size bins and simple residual relationships; report uncertainty rather than presenting one pooled coefficient. Treat constant inputs as undefined. Repository resampling can show sensitivity; do not treat nested functions as independent repositories. - -No live provider is needed. Local source samples are copied from authorized pinned checkouts; failures come from synthetic missing, constant, overlapping-scope and outlier rows. Rollback removes the evaluation script and report additions without changing analyzer behavior. - -## Milestone 1: The evaluation catches misleading metric relationships - -Create deterministic cases: same CCN with different nesting, same size with different branching, CRAP with identical components, unsupported zeros, constant fields, and one rare useful exception among strongly related counts. - -Produce a compact Markdown/JSON comparison showing associations and differing target queues. Reconcile 32 API fields, 27 report definitions and 22 selectors against the baseline inventory. Extend the inventory explicitly if metric trust changes identifiers. - -### Desired end state - -- Results reproduce from the same manifest and seed. -- Missing and constant metrics never appear as independent useful signals. -- A rare useful exception survives an otherwise high correlation. - -## Milestone 2: Three repositories yield a reviewable portfolio decision - -Select Srcly, one different TS/JS repository, and one Python repository with authorization. Pin revisions, analyzer versions, generated/vendor/test classifications and exclusions. Keep files, functions and folders separate. Do not pool language-specific unsupported measurements. - -Compare CCN plus size, the proposed compact catalog, and the full organized catalog. Review up to 20 deduplicated targets per arm per repository, starting with a smaller balanced subset. Shuffle metric labels and order where feasible. Record Byron's next action, its source evidence, time, and whether another signal already supplied it. Human review is a required execution observation; automated review may prepare cases but cannot claim Byron's judgment. - -Give each metric and candidate a retain, optional, investigate, defer or reject outcome. Require a candidate default to add a justified decision in at least two repositories, with an explicit false-positive and cost account. This is a provisional product gate, not statistical significance. Evaluate cognitive complexity on saved or externally supplied values before implementing a new analyzer. - -### Desired end state - -- A manifest, comparison tables and adjudicated cases explain every disposition. -- The compact default can be rejected if it loses useful decisions. -- No empirical claim exceeds the pilot's actual support and language coverage. - -## Milestone 3: Apply the proven catalog choice - -Update `metricsStore.tsx`, `MetricPicker.tsx`, report definitions/profiles, and documentation only for choices supported by the pilot. Retain neutral diagnostics where useful. Apply a coherent API change directly if removal is justified; do not add old/new schema coexistence. - -Browser checks prove that retained measurements are discoverable and that the default queue leads to source. Run client tests/build and relevant reporting tests. Keep the pilot artifact as the decision record. - -### Desired end state - -- UI and headless defaults match the recorded portfolio decision. -- Reverting the catalog commit restores the prior choice without a data migration. - -## Below the cut line - -- Twelve-repository expansion until the pilot earns it. -- Default Maintainability Index, Halstead effort/bug forecasts, centrality-as-quality and people scoring. -- Automated feature selection, universal defect thresholds and hosting infrastructure. diff --git a/docs/intent/metric-portfolio/initiative-map.json b/docs/intent/metric-portfolio/initiative-map.json deleted file mode 100644 index 9feb020b..00000000 --- a/docs/intent/metric-portfolio/initiative-map.json +++ /dev/null @@ -1,267 +0,0 @@ -{ - "schema_version": "0.5.0", - "initiative": { - "title": "A metric portfolio that earns its attention", - "slug": "metric-portfolio", - "scale": "capability", - "current_phase": "plan", - "last_updated": "2026-10-15", - "summary": "Select a compact portfolio using stratified association and source-backed decision cases.", - "status": "planned", - "next_action": "After metric-trust milestone 2, prove evaluation behavior with deliberate counterexamples." - }, - "artifacts": { - "intent_brief": { - "path": "intent-brief.md", - "status": "current", - "updated_at": "2026-10-15" - }, - "brainstorm": { - "path": "brainstorm.md", - "status": "not-requested", - "updated_at": null - }, - "shape_brief": { - "path": "shape-brief.md", - "status": "current", - "updated_at": "2026-10-15" - }, - "implementation_plan": { - "path": "implementation-plan.md", - "status": "current", - "updated_at": "2026-10-15" - } - }, - "sources": [ - { - "id": "user", - "type": "conversation", - "location": "Current Codex chat", - "anchor": "2026-10-15 original unique-information request and full-process approval", - "authority": "user" - }, - { - "id": "research", - "type": "research", - "location": "docs/research/metrics-2026-10-15/originals/recommendations.md", - "anchor": "R08, R09, R17", - "authority": "external proposals checked against baseline; not product intent" - }, - { - "id": "repo", - "type": "repository", - "location": "server/app/services/reporting.py; client/src/utils/metricsStore.tsx; client/src/components/MetricPicker.tsx", - "anchor": "83dcbf40879ca06ab07cdfcf365b5c0ccb7608a4", - "authority": "current source" - } - ], - "intent": { - "claims": [ - { - "id": "C1", - "statement": "Provide unique actionable information per metric.", - "kind": "outcome", - "commitment": "explicit", - "importance": "core", - "status": "active", - "time_horizon": "first release", - "sources": [ - "user" - ], - "confidence": "high" - }, - { - "id": "C2", - "statement": "Avoid several highly correlated metrics that add little information.", - "kind": "anti-goal", - "commitment": "explicit", - "importance": "core", - "status": "active", - "time_horizon": "first release", - "sources": [ - "user" - ], - "confidence": "high" - }, - { - "id": "C3", - "statement": "Review current metrics and the field including CRAP before proposing initiatives.", - "kind": "sequence", - "commitment": "explicit", - "importance": "core", - "status": "active", - "time_horizon": "proof", - "sources": [ - "user" - ], - "confidence": "high" - }, - { - "id": "C4", - "statement": "Research and planning are authorized; application implementation remains planned.", - "kind": "constraint", - "commitment": "explicit", - "importance": "core", - "status": "active", - "time_horizon": "proof", - "sources": [ - "user" - ], - "confidence": "high" - }, - { - "id": "C5", - "statement": "Select a compact portfolio using stratified association and source-backed decision cases.", - "kind": "implementation hypothesis", - "commitment": "derived", - "importance": "supporting", - "status": "active", - "time_horizon": "proof", - "sources": [ - "research", - "repo" - ], - "confidence": "medium" - } - ], - "themes": [], - "conflicts": [], - "human_questions": [] - }, - "brainstorm": { - "status": "not-requested", - "requested_count": 7, - "options": [], - "selected_option": null - }, - "shape": { - "status": "current", - "appetite": "Small isolated local proofs; no enterprise infrastructure or compatibility layers.", - "problem": "Select a compact portfolio using stratified association and source-backed decision cases.", - "recommended_shape": "A metric portfolio that earns its attention", - "success_moves": [ - "Use deterministic observation artifacts before scanning a broad corpus.", - "Start with three repositories and stop before twelve unless useful.", - "Compare compact and full catalogs so the favored approach can lose.", - "Keep Byron adjudication separate from automated case preparation." - ], - "first_proof": { - "question": "Does evaluation distinguish mathematical overlap, missing support and useful exceptions?", - "artifact": "Deterministic comparison Markdown/JSON with constant and equal-CCN cases.", - "decision_rule": "Reject the method if missing data earns independence or high correlation alone removes a useful exception." - }, - "rabbit_holes": [], - "open_decisions": [] - }, - "implementation": { - "status": "current", - "repository_snapshot": { - "root": "/Users/byronwall/.codex/worktrees/46c0/srcly", - "revision": "83dcbf40879ca06ab07cdfcf365b5c0ccb7608a4", - "worktree_status": "Research/planning documents only; no application implementation.", - "subsystems": [ - "server/app/services/reporting.py", - "client/src/utils/metricsStore.tsx", - "client/src/components/MetricPicker.tsx" - ], - "test_commands": [ - "cd server && uv run python scripts/evaluate_metric_portfolio.py --manifest ../tmp/portfolio/manifest.json --out ../tmp/portfolio/results", - "cd client && pnpm test", - "cd client && pnpm build" - ], - "deployment_surfaces": [] - }, - "dependencies": [ - { - "id": "corpus", - "name": "Corrected corpus observations and human decisions", - "material": true, - "application_seam": "Standalone evaluation input manifest and saved observations", - "first_proof_role": { - "mode": "local-substitute", - "rationale": "Deterministic counterexamples can reject the analysis method before any real corpus or human review." - }, - "failure_modes": [ - "constant series", - "unsupported measurements", - "overlapping populations", - "outlier dominance" - ], - "proof_rungs": [ - "local-fixture", - "contract-test", - "real-local-output" - ], - "local_proof": "Fixed observation fixtures with known useful exceptions and missingness.", - "live_proof": "Three explicitly authorized pinned repositories and Byron action review; no hosted system.", - "rollout": "Optional input or coherent commit; omit/revert without parallel API support." - } - ], - "fast_local_loop": { - "command": "cd server && uv run python scripts/evaluate_metric_portfolio.py --manifest ../tmp/portfolio/manifest.json --out ../tmp/portfolio/results", - "network_required": false, - "shared_infrastructure_required": false, - "failure_injection": [ - "constant series", - "unsupported measurements", - "overlapping populations", - "outlier dominance" - ], - "status": "Proposed commands for implementation; new scripts/tests do not exist yet." - }, - "milestones": [ - { - "id": "m1", - "title": "The evaluation catches misleading metric relationships", - "outcome": "A deterministic report reveals undefined associations, support rates and useful exceptions.", - "dependencies_introduced": [], - "proof": "Known equal-size/equal-CCN, constant, unavailable and rare-exception cases reproduce.", - "valid_intermediate_state": "A deterministic report reveals undefined associations, support rates and useful exceptions. Existing source-only scanning remains usable.", - "rollback": "Revert the scoped milestone commit or omit optional evidence; no stored-data migration.", - "status": "planned", - "evidence": [] - }, - { - "id": "m2", - "title": "Three repositories yield a reviewable portfolio decision", - "outcome": "A pinned three-repository pilot records decisions for compact and full portfolios.", - "dependencies_introduced": [], - "proof": "Manifest, size/language strata, rank overlap and shuffled human action review; no broad validity claims.", - "valid_intermediate_state": "A pinned three-repository pilot records decisions for compact and full portfolios. Existing source-only scanning remains usable.", - "rollback": "Revert the scoped milestone commit or omit optional evidence; no stored-data migration.", - "status": "planned", - "evidence": [] - }, - { - "id": "m3", - "title": "Apply the proven catalog choice", - "outcome": "The UI and reports adopt only supported catalog changes.", - "dependencies_introduced": [], - "proof": "Catalog decision record plus local browser discoverability, client build and reporting tests.", - "valid_intermediate_state": "The UI and reports adopt only supported catalog changes. Existing source-only scanning remains usable.", - "rollback": "Revert the scoped milestone commit or omit optional evidence; no stored-data migration.", - "status": "planned", - "evidence": [] - } - ], - "cut_line": [ - "Twelve-repository expansion until the pilot earns it.", - "Default Maintainability Index, Halstead effort/bug forecasts, centrality-as-quality and people scoring.", - "Automated feature selection, universal defect thresholds and hosting infrastructure." - ], - "open_decisions": [], - "initiative_dependencies": [ - { - "initiative": "metric-trust", - "gate": "milestone 2: reliable populations and units" - } - ] - }, - "history": [ - { - "date": "2026-10-15", - "change": "Created planned initiative from collected Pro research and independent source checks. Application proofs remain unrun." - } - ], - "closure": null -} diff --git a/docs/intent/metric-portfolio/intent-brief.md b/docs/intent/metric-portfolio/intent-brief.md deleted file mode 100644 index 654ea665..00000000 --- a/docs/intent/metric-portfolio/intent-brief.md +++ /dev/null @@ -1,32 +0,0 @@ -# A metric portfolio that earns its attention - -## My read - -Byron wants each metric to reveal something useful that other metrics miss. A large catalog can create the appearance of insight while repeating size or branching in several forms. The intended result is a clear set of review decisions, not the largest number of counters or the lowest pairwise correlations. - -This initiative establishes which measurements deserve default attention, optional detail, or no place in Srcly. It compares the proposed compact portfolio with a full organized catalog. Both remain valid candidates until evidence separates them. The study must allow the compact proposal to lose when it hides important cases. - -The first audience is Byron reviewing personal repositories locally. A small study can reveal bad definitions, obvious overlap, and useful exceptions without claiming universal statistical validity. Broader language claims need later evidence. The work should make additions cheaper to reject before they become permanent UI and maintenance obligations. - -## What matters most - -- Separate shared formulas, observed association, and distinct decision value. -- Control for size, language, scope and unavailable analysis. -- Keep useful exceptions even when metrics correlate strongly. -- Reject noisy metrics even when their correlations are low. - -## Boundaries - -Begin after metric trust provides reliable populations and support. Use three explicitly selected local or public repositories at pinned revisions. Do not search private folders for research subjects or run their test configurations automatically. - -Do not train a defect predictor, estimate remediation hours, or promise a statistically representative result. A proposed twelve-repository extension is conditional on the pilot being useful. No metric is removed solely because a coefficient crosses a threshold. - -## What seems settled - -CCN, nesting, average function size and several counts overlap conceptually. That does not establish measured redundancy. CRAP is exactly derived from complexity and coverage, but its presentation could still improve a decision. These are different questions. - -The research supplies definitions, candidate proofs and counterexamples. It does not supply repository correlations or product tests. Current API/report/selector coverage must remain traceable throughout the evaluation. - -## Next step after confirmation - -Build a small reproducible comparison artifact from corrected measurements. Use deliberate equal-size and equal-CCN examples before the repository pilot. Then ask Byron to judge a small shuffled set of source-backed decisions. The [plan](implementation-plan.md) separates that human observation from data collection. All execution remains planned. diff --git a/docs/intent/metric-portfolio/shape-brief.md b/docs/intent/metric-portfolio/shape-brief.md deleted file mode 100644 index cdbb08c6..00000000 --- a/docs/intent/metric-portfolio/shape-brief.md +++ /dev/null @@ -1,42 +0,0 @@ -# A useful metric portfolio — shape - -**Outcome:** Default metrics earn their attention through distinct review decisions. - -**Primary flow:** Load comparable observations → compare portfolios → inspect differing targets → retain useful signals and document exclusions. - -## Feature scope - -```text -Portfolio evaluation -├── PLANNED ADDITIONS -│ ├── Reproducible three-repository pilot -│ │ ├── Revision, analyzer, exclusions and support recorded -│ │ └── Separate language, file/function and size groups -│ ├── Association and decision comparison -│ │ ├── Rank overlap, size-adjusted relationships and missing rates -│ │ └── Shuffled source cases with useful and rejected outcomes -│ └── Recommended catalog -│ ├── Compact defaults versus organized full catalog -│ └── Each candidate has a reason, cost and revisit condition -└── EVALUATE BEFORE COMMITTING - ├── Cognitive complexity beyond CCN, nesting and size - └── Wider corpus only when the pilot yields useful distinctions -``` - -## Decisions and boundaries - -**Appetite:** Three repositories and a small review sample first. Use existing scan artifacts and a standalone analysis script. - -**Key decision:** Review changes in action as well as correlations. A high association cannot erase a rare useful case. A low association cannot excuse broken support or noise. - -**Boundary:** No feature-selection platform, model training, or opaque composite. Review time and setup cost count against a candidate. The existing catalog remains accessible during evaluation; no compatibility layer is required. - -**First proof:** A repeatable comparison artifact with deliberate counterexamples. - -**Try:** Compare equal-size flat/deep functions and a constant or mostly unavailable metric before collecting a repository cohort. - -**Observe:** Check undefined associations, support counts, queue overlap, and which source case changes the proposed action. - -**Decide:** Reject the analysis method if it calls missing data independent or discards a useful case solely for high correlation. - -See the [implementation plan](implementation-plan.md). diff --git a/docs/research/metrics-2026-10-15/README.md b/docs/research/metrics-2026-10-15/README.md index c7587bc7..db12cd01 100644 --- a/docs/research/metrics-2026-10-15/README.md +++ b/docs/research/metrics-2026-10-15/README.md @@ -9,7 +9,7 @@ The research covers 32 API fields, 27 report definitions, 22 selectable metrics, | Initiative | Result | First proof | | --- | --- | --- | | [Metric trust](https://github.com/byronwall/srcly/blob/0bfb05ba/docs/intent/metric-trust/shape-brief.md) | Correct scope, units, support, rollups, and ranking | A measured zero differs from unavailable data in one real scan and report | -| [Portfolio evaluation](../../intent/metric-portfolio/shape-brief.md) | Evidence for a compact default and useful diagnostics | Three pinned repositories reveal retained and rejected decision cases | +| [Portfolio evaluation](https://github.com/byronwall/srcly/blob/0bfb05ba/docs/intent/metric-portfolio/shape-brief.md) | Evidence for a compact default and useful diagnostics | Three pinned repositories reveal retained and rejected decision cases | | [Git activity](../../intent/git-activity/shape-brief.md) | Explain why difficult code deserves attention now | Verified file histories change a justified priority decision | | [Imported quality evidence](../../intent/imported-quality-evidence/shape-brief.md) | Inspect cycles, clones, diagnostics, and test gaps | An existing graph edge leads to its original source declaration |