diff --git a/CHANGELOG.md b/CHANGELOG.md index be41274..81488d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,9 +8,12 @@ All notable changes to DebtLens are documented here. This project adheres to ### Added - `debtlens history record` / `history show` with `.debtlens/history.jsonl` ledger and timeline reports. -- Payoff ranking via `--sort payoff`, JSON `payoffScore`, and top-payoff report sections. -- `debtlens calibrate` for percentile-based threshold suggestions with optional `--write`. -- Interactive `debtlens triage` for keep/baseline/suppress workflows. +- Payoff ranking via `--sort payoff`, JSON `payoffScore`, and top-payoff report sections + ([#251](https://github.com/ColumbusLabs/DebtLens/issues/251)). +- `debtlens calibrate` for percentile-based threshold suggestions with optional `--write` + ([#256](https://github.com/ColumbusLabs/DebtLens/issues/256)). +- Interactive `debtlens triage` for keep/baseline/suppress workflows + ([#255](https://github.com/ColumbusLabs/DebtLens/issues/255)). - `debtlens fix` dry-run autofix allowlist for duplicated literals. - Opt-in `feature-flags` pack with `stale-feature-flag` detector. - `--concurrency` and `--cache-dir` scan controls for large-repo performance. @@ -22,7 +25,8 @@ All notable changes to DebtLens are documented here. This project adheres to ([#260](https://github.com/ColumbusLabs/DebtLens/issues/260), [#261](https://github.com/ColumbusLabs/DebtLens/issues/261), [#262](https://github.com/ColumbusLabs/DebtLens/issues/262)). -- Config `budgets` block and `debtlens scan --budget-report` for per-area debt SLO gating. +- Config `budgets` block and `debtlens scan --budget-report` for per-area debt SLO gating + ([#252](https://github.com/ColumbusLabs/DebtLens/issues/252)). - `debtlens scan --format badge` emits a self-contained SVG badge plus shields.io endpoint JSON ([#265](https://github.com/ColumbusLabs/DebtLens/issues/265)). - Scan results now warn when matched files exceed `maxFiles`; terminal output prints the diff --git a/README.md b/README.md index 2344cda..cc39b2e 100644 --- a/README.md +++ b/README.md @@ -191,6 +191,8 @@ debtlens init --pack core # starter config using the core rule pack preset debtlens init --policy @org/debtlens-policy # starter config from a shared policy package debtlens init --from-eslint eslint.config.json # print a migration suggestion without writing debtlens adopt # adoption report (dry run; recommends minSeverity) +debtlens calibrate . # suggest percentile-based threshold overrides +debtlens triage . # interactively keep, baseline, or suppress findings debtlens watch examples/react --rules todo-comment # rescan on file changes debtlens completions zsh # print shell completions debtlens mcp # stdio MCP server for Cursor/Claude-style agents @@ -343,6 +345,39 @@ The second command writes `debtlens.config.json` and `debtlens-baseline.json` (b For serious open-source repositories and broad monorepos, treat adoption as a scoped rollout instead of a whole-repo gate on day one. Run `adopt` first, then start with `--changed origin/main` or a maintained source subdirectory, add `--package` for workspaces, keep generated/dependency outputs excluded, and narrow expensive or noisy rules with `--rules`, thresholds, baselines, or confidence floors before widening coverage. If the default 2,000-file cap appears, either raise it with `--max-files` for a deliberate full scan or make the target more precise with `--package`, `--include`, `--exclude`, `--changed`, or `--respect-gitignore`. +### Tune, prioritize, triage, and budget + +After `adopt` picks rules, use these commands to finish a low-noise rollout: + +```bash +# 1. Tune numeric thresholds to your repo's distributions +debtlens calibrate . --percentile 90 +debtlens calibrate . --percentile 90 --write # merge into debtlens.config.json + +# 2. Review highest-ROI findings first (enable hotspots when git history is available) +debtlens scan . --sort payoff --hotspots + +# 3. Walk the backlog interactively (requires a TTY; use --dry-run to preview) +debtlens triage . + +# 4. Protect cleaned areas with per-path budgets in debtlens.config.json +debtlens scan . --budget-report +debtlens scan . --fail-on high # budget breaches also fail the gate +``` + +Example `budgets` block: + +```json +{ + "budgets": { + "src/payments": { "maxIssues": 20, "maxHigh": 0 }, + "src/legacy": { "maxIssues": 250 } + } +} +``` + +See [`docs/prioritization.md`](./docs/prioritization.md) for the payoff formula, `priority` weights, and budget glob matching. + Named quality-gate presets give teams a shared rollout vocabulary: | Preset | Use when | Default behavior | diff --git a/docs/false-positives.md b/docs/false-positives.md index 6381c34..492d8ae 100644 --- a/docs/false-positives.md +++ b/docs/false-positives.md @@ -14,6 +14,35 @@ adding inline suppressions. | Low-confidence finding family | `ruleConfidenceFloors` or `--fail-on-confidence` | Keeps findings visible while preventing weak gates. | | One documented exception | Inline suppression with a reason | Auditable, local, and visible in JSON/SARIF output. | +## Calibrate thresholds to your repo + +Default thresholds are deliberately generic. On a large legacy repo they can be noisy; on a small repo they can miss real debt. After `debtlens adopt` picks rules and severity, run calibration to tune the numbers: + +```bash +# Preview percentile-based threshold suggestions (default p90) +debtlens calibrate . + +# Tune aggressiveness and merge suggestions into debtlens.config.json +debtlens calibrate . --percentile 85 --write +``` + +Calibration temporarily lowers supported numeric trigger thresholds so the sample +includes below-threshold code, not just existing findings. The report also lists +selected policy floors, boolean switches, similarity controls, and safety caps +under **Not calibrated** when they cannot be inferred honestly from a distribution. + +Calibration scans the target, collects observed metrics for threshold-driven rules (function length, branch counts, and similar), and suggests values at the chosen percentile so roughly the worst N% is flagged. Review the printed config snippet before using `--write`; unrelated config keys are preserved. + +Pair calibration with payoff ranking and triage for a low-noise first rollout: + +```bash +debtlens calibrate . --percentile 90 +debtlens scan . --sort payoff --hotspots +debtlens triage . +``` + +See [`docs/prioritization.md`](./prioritization.md) for payoff scoring and per-area budgets. + ## Baseline before suppressing ```bash diff --git a/docs/prioritization.md b/docs/prioritization.md new file mode 100644 index 0000000..7fa43db --- /dev/null +++ b/docs/prioritization.md @@ -0,0 +1,90 @@ +# Payoff prioritization + +DebtLens can rank findings by **payoff score** so teams fix the debt that costs the most first instead of working through a flat severity list. + +## When scores appear + +Each issue gets a `payoffScore` in JSON output when payoff ranking is enabled: + +- pass `--sort payoff` on `debtlens scan`, or +- enable git churn hotspots with `--hotspots` on the CLI, or +- pass `--blame-age` to include age in the score. + +Terminal, Markdown, and HTML reports include a **Top payoff targets** section when any issue carries a score. + +## Scoring formula + +Payoff combines signals already present on each finding: + +``` +payoffScore = severityWeight × confidence × churnFactor × ageFactor +``` + +| Factor | Default behavior | +| --- | --- | +| `severityWeight` | high 16, medium 8, low 3, info 1 | +| `confidence` | issue confidence, floored at 0.35 | +| `churnFactor` | `1 + log2(1 + churnMetric) × churnWeight` when hotspot data exists; otherwise 1 | +| `ageFactor` | `1 + min(introducedDaysAgo / 365, 2) × ageWeight` when blame age is available; otherwise 1 | + +Churn metrics come from [`src/core/hotspots.ts`](../src/core/hotspots.ts). Age uses optional `introducedDaysAgo` from `--blame-age`. + +## CLI usage + +```bash +# Sort findings by payoff and show top targets in the terminal report +debtlens scan . --sort payoff --hotspots + +# JSON consumers read payoffScore and a bounded top-target shortlist +debtlens scan . --sort payoff --format json +``` + +When payoff scores are enabled, JSON output also includes +`summary.topPayoffTargets`. It contains at most 10 compact targets ordered +deterministically by score, file, line, rule, and fingerprint. Full issue details +remain in `issues`. + +Use `--hotspots` when git history is available so churn boosts files that change often. In CI, check out enough history (`fetch-depth: 0` or a bounded `--churn-range`). + +## Config weights + +Tune weights under the `priority` block in `debtlens.config.json`: + +```json +{ + "priority": { + "severity": { "high": 20, "medium": 10, "low": 4, "info": 1 }, + "churn": 1.2, + "age": 0.75 + } +} +``` + +Higher `churn` and `age` weights amplify those factors. Severity weights follow the same keys as built-in severities. + +## Adoption workflow + +Payoff ranking fits between calibration and triage: + +1. `debtlens calibrate` — tune thresholds to your repo (see [`docs/false-positives.md`](./false-positives.md)). +2. `debtlens scan . --sort payoff --hotspots` — review the highest-ROI findings first. +3. `debtlens triage` — baseline or suppress the backlog interactively. +4. Add `budgets` — cap debt per directory after cleanup (see below). + +## Per-area budgets + +After triage, protect cleaned areas with path budgets: + +```json +{ + "budgets": { + "src/payments": { "maxIssues": 20, "maxHigh": 0 }, + "src/legacy": { "maxIssues": 250 } + } +} +``` + +- Budget breaches fail the scan gate (exit code 1) like `--fail-on`. +- `debtlens scan --budget-report` prints used/budget/headroom without failing — useful in CI dashboards. + +Keys are path globs (`src/payments/**`, `src/**/*.ts`, or exact prefixes). diff --git a/package-lock.json b/package-lock.json index c08a27f..146af98 100644 --- a/package-lock.json +++ b/package-lock.json @@ -12,6 +12,7 @@ "chokidar": "^4.0.3", "commander": "^15.0.0", "fast-glob": "^3.3.3", + "micromatch": "^4.0.8", "ts-morph": "^28.0.0", "yaml": "^2.9.0" }, diff --git a/package.json b/package.json index c51af26..b5c0c5d 100644 --- a/package.json +++ b/package.json @@ -75,6 +75,7 @@ "chokidar": "^4.0.3", "commander": "^15.0.0", "fast-glob": "^3.3.3", + "micromatch": "^4.0.8", "ts-morph": "^28.0.0", "yaml": "^2.9.0" }, diff --git a/schema/debtlens.scan-result.schema.json b/schema/debtlens.scan-result.schema.json index 6f65924..45b73c9 100644 --- a/schema/debtlens.scan-result.schema.json +++ b/schema/debtlens.scan-result.schema.json @@ -376,6 +376,73 @@ "type": "integer", "minimum": 0 }, + "topPayoffTargets": { + "type": "array", + "maxItems": 10, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "fingerprint", + "ruleId", + "file", + "severity", + "payoffScore" + ], + "properties": { + "id": { + "type": "string" + }, + "fingerprint": { + "type": "string" + }, + "ruleId": { + "type": "string" + }, + "file": { + "type": "string" + }, + "severity": { + "enum": [ + "info", + "low", + "medium", + "high" + ] + }, + "payoffScore": { + "type": "number", + "minimum": 0 + }, + "location": { + "type": "object", + "additionalProperties": false, + "required": [ + "startLine" + ], + "properties": { + "startLine": { + "type": "integer", + "minimum": 1 + }, + "startColumn": { + "type": "integer", + "minimum": 1 + }, + "endLine": { + "type": "integer", + "minimum": 1 + }, + "endColumn": { + "type": "integer", + "minimum": 1 + } + } + } + } + } + }, "warnings": { "type": "array", "items": { diff --git a/src/cli/adoptionThresholds.ts b/src/cli/adoptionThresholds.ts index 16c8eb3..716f572 100644 --- a/src/cli/adoptionThresholds.ts +++ b/src/cli/adoptionThresholds.ts @@ -9,50 +9,141 @@ export interface ThresholdSuggestion { observedValues?: number[]; } -interface EvidenceThreshold { +export interface CalibrationMetric { key: string; + ruleIds: readonly string[]; + source: "evidence" | "message"; pattern: RegExp; - ruleId?: string; } -const evidenceThresholds: EvidenceThreshold[] = [ - { key: "large-component.maxLines", pattern: /^Lines: (\d+) \// }, - { key: "large-component.maxHooks", pattern: /^Hook calls: (\d+) \// }, - { key: "large-component.maxBranches", pattern: /^Branch points: (\d+) \// }, - { key: "effect-complexity.maxLines", pattern: /^Lines: (\d+) \//, ruleId: "effect-complexity" }, - { key: "effect-complexity.maxDependencies", pattern: /^Dependencies: (\d+) \// }, +export interface CalibrationDiagnostic { + key: string; + ruleIds: readonly string[]; + reason: string; +} + +const metric = (key: string, ruleIds: string | readonly string[], source: "evidence" | "message", pattern: RegExp): CalibrationMetric => ({ + key, ruleIds: typeof ruleIds === "string" ? [ruleIds] : ruleIds, source, pattern, +}); + +export const calibrationMetrics: readonly CalibrationMetric[] = [ + metric("large-component.maxLines", "large-component", "evidence", /^Lines: (\d+)\b/), + metric("large-component.maxHooks", "large-component", "evidence", /^Hook calls: (\d+)\b/), + metric("large-component.maxBranches", "large-component", "evidence", /^Branch points: (\d+)\b/), + metric("large-function.maxLines", ["large-function", "python-large-function", "kotlin-large-function", "swift-large-function", "ruby-large-function"], "evidence", /^Lines: (\d+)\b/), + metric("large-function.maxBranches", ["large-function", "python-large-function", "kotlin-large-function", "swift-large-function", "ruby-large-function"], "evidence", /^Branch points: (\d+)\b/), + metric("effect-complexity.maxLines", "effect-complexity", "evidence", /^Lines: (\d+)\b/), + metric("effect-complexity.maxDependencies", "effect-complexity", "evidence", /^Dependencies: (\d+)\b/), + metric("state-sprawl.maxStatefulHooks", "state-sprawl", "message", /manages (\d+) stateful hook calls/), + metric("prop-drilling.maxForwardedProps", "prop-drilling", "message", /forwards (\d+) props/), + metric("context-provider-sprawl.maxProviders", "context-provider-sprawl", "message", /wraps children in (\d+) distinct/), + metric("rn-host-forwarding.maxForwardedProps", "rn-host-forwarding", "message", /forwards (\d+) wrapper props/), + metric("rn-host-forwarding.maxHostTargets", "rn-host-forwarding", "message", /into (\d+) host/), + metric("route-handler-size.maxLines", "route-handler-size", "evidence", /^Lines: (\d+)\b/), + metric("route-handler-size.maxBranches", "route-handler-size", "evidence", /^Branch points: (\d+)\b/), + metric("route-handler-size.maxAwaits", "route-handler-size", "evidence", /^Await expressions: (\d+)\b/), + metric("data-loader-sprawl.maxLines", "data-loader-sprawl", "evidence", /^Lines: (\d+)\b/), + metric("data-loader-sprawl.maxBranches", "data-loader-sprawl", "evidence", /^Branch points: (\d+)\b/), + metric("data-loader-sprawl.maxAwaits", "data-loader-sprawl", "evidence", /^Await expressions: (\d+)\b/), + metric("data-loader-sprawl.maxFetches", "data-loader-sprawl", "evidence", /^Fetch calls: (\d+)\b/), + metric("handler-depth.maxDepth", "handler-depth", "evidence", /^(?:Control depth|Nested callbacks): (\d+)\b/), + metric("handler-depth.maxMiddleware", "handler-depth", "evidence", /^Middleware arguments: (\d+)\b/), + metric("route-sprawl.maxRoutes", "route-sprawl", "message", /registers (\d+) routes/), + metric("python-route-sprawl.maxRoutes", "python-route-sprawl", "message", /registers (\d+) routes/), + metric("rails-route-sprawl.maxRoutes", "rails-route-sprawl", "message", /registers (\d+) Rails routes/), + metric("rails-controller-sprawl.maxActions", "rails-controller-sprawl", "message", /declares (\d+) public controller actions/), + metric("barrel-file.maxReExports", "barrel-file", "message", /with (\d+) (?:re-)?exports?/), + metric("api-surface-sprawl.maxExports", "api-surface-sprawl", "evidence", /^Exports: (\d+)\b/), + metric("complex-control-flow.maxComplexity", ["complex-control-flow", "python-complex-control-flow"], "evidence", /^(?:Complexity score|Cyclomatic complexity): (\d+)\b/), + metric("complex-control-flow.maxDepth", ["complex-control-flow", "python-complex-control-flow"], "evidence", /^(?:Max nesting depth|Control-flow depth): (\d+)\b/), + metric("cognitive-complexity.max", "cognitive-complexity", "evidence", /^Cognitive complexity: (\d+)\b/), + metric("long-parameter-list.maxParams", "long-parameter-list", "evidence", /^Parameters: (\d+)\b/), + metric("long-parameter-list.maxBooleans", "long-parameter-list", "evidence", /^Boolean parameters: (\d+)\b/), + metric("god-file.maxLines", "god-file", "evidence", /^Lines: (\d+)\b/), + metric("god-file.maxExports", "god-file", "evidence", /^Exports: (\d+)\b/), + metric("god-file.maxTopLevelDecls", "god-file", "evidence", /^Top-level declarations: (\d+)\b/), + metric("naming-drift.minVariants", "naming-drift", "message", /uses (\d+) competing terms/), + metric("vue-large-script.maxLines", "vue-large-script", "evidence", /^Script lines: (\d+)\b/), + metric("vue-large-script.maxFunctionLines", "vue-large-script", "evidence", /^Lines: (\d+)\b/), + metric("vue-large-script.maxBranches", "vue-large-script", "evidence", /^Branch points: (\d+)\b/), + metric("svelte-large-script.maxLines", "svelte-large-script", "evidence", /^Script lines: (\d+)\b/), + metric("svelte-large-script.maxFunctionLines", "svelte-large-script", "evidence", /^Lines: (\d+)\b/), + metric("svelte-large-script.maxBranches", "svelte-large-script", "evidence", /^Branch points: (\d+)\b/), + ...["compose-large-composable", "swiftui-large-view"].flatMap((ruleId) => [ + metric(`${ruleId}.maxLines`, ruleId, "evidence", /(?:^|: )(\d+) lines\b/i), + metric(`${ruleId}.maxBranches`, ruleId, "evidence", /(\d+) branch points\b/i), + ]), + metric("compose-large-composable.maxLocalState", "compose-large-composable", "evidence", /(\d+) local state holders\b/i), + metric("swiftui-large-view.maxLocalState", "swiftui-large-view", "evidence", /(\d+) local state holders\b/i), + metric("compose-state-hoisting.maxLocalState", "compose-state-hoisting", "message", /owns (\d+) local/), + metric("swiftui-state-sprawl.maxStateHolders", "swiftui-state-sprawl", "message", /owns (\d+) local/), ] as const; -export function buildThresholdSuggestions(result: ScanResult, options: ScanOptions): ThresholdSuggestion[] { - const observed = new Map(); +const diagnostic = (key: string, ruleIds: string | readonly string[], reason: string): CalibrationDiagnostic => ({ + key, ruleIds: typeof ruleIds === "string" ? [ruleIds] : ruleIds, reason, +}); + +const duplicateRules = ["duplicate-logic", "python-duplicate-logic", "kotlin-duplicate-logic", "swift-duplicate-logic", "ruby-duplicate-logic"] as const; +const deadRules = ["dead-abstraction", "python-dead-abstraction", "kotlin-dead-abstraction", "swift-dead-abstraction", "ruby-dead-abstraction"] as const; +export const calibrationDiagnostics: readonly CalibrationDiagnostic[] = [ + diagnostic("god-file.minAxes", "god-file", "minimum multi-axis trigger is a rule policy, not a repository distribution"), + ...["minSimilarity", "minStructuralSimilarity", "minLines", "maxSnippets"].map((name) => diagnostic(`duplicate-logic.${name}`, duplicateRules, "similarity/corpus controls cannot be inferred from emitted findings")), + ...["minSimilarity", "minStructuralSimilarity", "minLines"].map((name) => diagnostic(`test-duplication.${name}`, "test-duplication", "similarity controls cannot be inferred from emitted findings")), + diagnostic("dead-abstraction.maxWrapperLines", deadRules, "wrapper size is only meaningful after semantic wrapper classification"), + diagnostic("duplicated-literal.minLength", "duplicated-literal", "minimum token length is a policy floor"), + diagnostic("duplicated-literal.minCount", "duplicated-literal", "minimum repetition count is a policy floor"), + diagnostic("config-drift.maxConfigFiles", "config-drift", "repository safety cap is not an observed finding metric"), + diagnostic("import-cycle.minCycleSize", "import-cycle", "minimum cycle size is a policy floor"), + diagnostic("import-cycle.allowTypeOnly", "import-cycle", "boolean behavior switch is not calibratable"), + diagnostic("weak-test-boundary.allowTypeOnly", "weak-test-boundary", "boolean behavior switch is not calibratable"), + diagnostic("empty-catch.allowCommentOnly", ["empty-catch", "python-error-handling", "kotlin-empty-catch"], "boolean behavior switch is not calibratable"), + diagnostic("floating-promise.allowVoid", "floating-promise", "boolean behavior switch is not calibratable"), + diagnostic("floating-promise.maxPerFile", "floating-promise", "per-file reporting cap is not a trigger distribution"), + diagnostic("commented-out-code.minLines", "commented-out-code", "minimum block size is a policy floor"), + diagnostic("commented-out-code.maxPerFile", "commented-out-code", "per-file reporting cap is not a trigger distribution"), + diagnostic("ai-instruction-duplication.maxInstructionFiles", "ai-instruction-duplication", "repository safety cap is not an observed finding metric"), + diagnostic("ai-instruction-duplication.minBlockLength", "ai-instruction-duplication", "minimum text length is a policy floor"), + diagnostic("ai-instruction-contradiction.maxInstructionFiles", "ai-instruction-contradiction", "repository safety cap is not an observed finding metric"), + diagnostic("ai-instruction-contradiction.minBlockLength", "ai-instruction-contradiction", "minimum text length is a policy floor"), +] as const; +export const calibrationThresholdKeys = calibrationMetrics.map((entry) => entry.key); + +export function calibrationThresholdOverrides(selectedRules?: readonly string[]): Record { + const selected = selectedRules ? new Set(selectedRules) : undefined; + const entries = calibrationMetrics.filter((entry) => !selected || entry.ruleIds.some((ruleId) => selected.has(ruleId))); + const overrides = Object.fromEntries(entries.map((entry) => [entry.key, 0])); + if (!selected || selected.has("god-file")) overrides["god-file.minAxes"] = 1; + return overrides; +} + +export function calibrationDiagnosticsForRules(selectedRules?: readonly string[]): CalibrationDiagnostic[] { + const selected = selectedRules ? new Set(selectedRules) : undefined; + return calibrationDiagnostics.filter((entry) => !selected || entry.ruleIds.some((ruleId) => selected.has(ruleId))); +} + +export function collectThresholdObservations(result: ScanResult): Map { + const observed = new Map(); for (const issue of result.issues) { - for (const evidence of issue.evidence ?? []) { - for (const threshold of evidenceThresholds) { - if (threshold.ruleId && threshold.ruleId !== issue.ruleId) continue; - if (!threshold.ruleId && threshold.key.startsWith("large-component.") && issue.ruleId !== "large-component") continue; - const match = evidence.match(threshold.pattern); - if (!match) continue; - pushObserved(observed, threshold.key, Number(match[1])); + for (const entry of calibrationMetrics) { + if (!entry.ruleIds.includes(issue.ruleId)) continue; + const values = entry.source === "message" ? [issue.message] : issue.evidence ?? []; + for (const value of values) { + const match = value.match(entry.pattern); + if (match?.[1]) pushObserved(observed, entry.key, Number(match[1])); } } - - if (issue.ruleId === "state-sprawl") { - const count = issue.message.match(/manages (\d+) stateful hook calls/)?.[1]; - if (count) pushObserved(observed, "state-sprawl.maxStatefulHooks", Number(count)); - } - if (issue.ruleId === "prop-drilling") { - const count = issue.message.match(/forwards (\d+) props/)?.[1]; - if (count) pushObserved(observed, "prop-drilling.maxForwardedProps", Number(count)); - } } + return observed; +} - return [...observed.entries()] +export function buildThresholdSuggestions(result: ScanResult, options: ScanOptions): ThresholdSuggestion[] { + return [...collectThresholdObservations(result).entries()] .map(([key, values]) => { - const current = options.thresholds[key]; + const current = options.thresholds[key] ?? 0; const observedP90 = percentile(values, 0.9); - const suggested = Math.max(Math.ceil(observedP90 * 1.1), Math.ceil(current ?? 0)); - return { key, current: current ?? 0, suggested, observedP90, samples: values.length, observedValues: [...values] }; + const suggested = Math.max(Math.ceil(observedP90 * 1.1), Math.ceil(current)); + return { key, current, suggested, observedP90, samples: values.length, observedValues: [...values] }; }) .filter((suggestion) => suggestion.current > 0 && suggestion.suggested > suggestion.current) .sort((left, right) => left.key.localeCompare(right.key)); diff --git a/src/cli/calibrate.ts b/src/cli/calibrate.ts index d81ab3b..7b1c64d 100644 --- a/src/cli/calibrate.ts +++ b/src/cli/calibrate.ts @@ -4,6 +4,7 @@ import { loadEffectiveConfig } from "../config/loadConfig.js"; import { mergeConfig } from "../config/mergeConfig.js"; import { mergeDebtLensConfig } from "../config/loadConfig.js"; import { buildCalibrateSuggestions, renderCalibrateReport } from "../core/calibrate.js"; +import { calibrationThresholdOverrides } from "./adoptionThresholds.js"; import { scan } from "../core/scan.js"; import { parseSeverity } from "../core/severity.js"; import { loadConfiguredPlugins } from "./scanPipeline.js"; @@ -36,7 +37,13 @@ export async function runCalibrate(input: CalibrateInput): Promise { pluginThresholds: pluginContribution?.thresholds, pluginVocabulary: pluginContribution?.vocabulary, }); - const result = await scan(options); + const result = await scan({ + ...options, + thresholds: { + ...options.thresholds, + ...calibrationThresholdOverrides(options.rules), + }, + }); const calibrate = buildCalibrateSuggestions(result, options, { percentile: input.percentile ?? 90, }); diff --git a/src/cli/commands/calibrate.ts b/src/cli/commands/calibrate.ts index 8b4cb9a..5da0b10 100644 --- a/src/cli/commands/calibrate.ts +++ b/src/cli/commands/calibrate.ts @@ -9,6 +9,9 @@ export function registerCalibrateCommand(program: Command): void { .argument("[target]", "directory or file to scan", ".") .option("--cwd ", "working directory", process.cwd()) .option("--config ", "path to debtlens.config.json") + .option("--pack ", "rule pack preset to scan with") + .option("--rules ", "comma-separated rule ids to run") + .option("--threshold ", "comma-separated key=value threshold overrides") .option("--percentile ", "percentile used for suggestions (50-99)", parseInteger) .option("--write", "merge suggested thresholds into debtlens.config.json") .action(async (target: string, rawOptions: Record) => { diff --git a/src/cli/commands/scan.ts b/src/cli/commands/scan.ts index 489154c..49b5b51 100644 --- a/src/cli/commands/scan.ts +++ b/src/cli/commands/scan.ts @@ -8,7 +8,7 @@ import { resolveWorkspacePackage } from "../../config/workspaces.js"; import { DEFAULT_BASELINE_FILENAME, createBaseline, writeBaseline } from "../../core/baseline.js"; import { evaluateBudgets, renderBudgetReport, type BudgetEvaluation } from "../../core/budgets.js"; import { buildOwnershipReport, renderOwnershipReportTerminal } from "../../core/ownershipReport.js"; -import { enrichIssuesWithPayoffScores, sortIssuesByPayoff } from "../../core/priority.js"; +import { enrichIssuesWithPayoffScores, sortIssuesByPayoff, topPayoffIssues } from "../../core/priority.js"; import { buildGitChurnHotspots } from "../../core/hotspots.js"; import { buildOwnershipSummary, loadCodeowners } from "../../core/ownership.js"; import { scan } from "../../core/scan.js"; @@ -457,6 +457,17 @@ function enrichPayoffScores( weights: fileConfig.priority, }); } + if (reported.issues.some((issue) => issue.payoffScore !== undefined)) { + reported.summary.topPayoffTargets = topPayoffIssues(reported.issues, 10).map((issue) => ({ + id: issue.id, + fingerprint: issue.fingerprint, + ruleId: issue.ruleId, + file: issue.file, + severity: issue.severity, + payoffScore: issue.payoffScore ?? 0, + ...(issue.location ? { location: issue.location } : {}), + })); + } } function computeScanExitCode(input: { diff --git a/src/cli/triage.ts b/src/cli/triage.ts index a2b3637..d19a162 100644 --- a/src/cli/triage.ts +++ b/src/cli/triage.ts @@ -2,8 +2,8 @@ import { createInterface } from "node:readline/promises"; import { resolve } from "node:path"; import { loadEffectiveConfig } from "../config/loadConfig.js"; import { mergeConfig } from "../config/mergeConfig.js"; -import { existsSync } from "node:fs"; -import { DEFAULT_BASELINE_FILENAME, createBaseline, loadBaseline, writeBaseline } from "../core/baseline.js"; +import { existsSync, readFileSync, statSync, writeFileSync } from "node:fs"; +import { DEFAULT_BASELINE_FILENAME, addIssuesToBaseline, createBaseline, filterIssues, loadBaseline, writeBaseline } from "../core/baseline.js"; import { scan } from "../core/scan.js"; import type { DebtIssue, ScanOptions } from "../core/types.js"; import { loadConfiguredPlugins } from "./scanPipeline.js"; @@ -20,6 +20,8 @@ export interface TriageInput { cliOptions?: Record; input?: NodeJS.ReadableStream; output?: NodeJS.WritableStream; + /** Injectable prompt for tests; defaults to readline when omitted. */ + ask?: (message: string) => Promise; } export interface TriageActionResult { @@ -49,25 +51,31 @@ export async function runTriage(input: TriageInput): Promise pluginVocabulary: pluginContribution?.vocabulary, }); const result = await scan(options); - const issues = [...result.issues]; const baselinePath = resolve(cwd, input.baselinePath ?? DEFAULT_BASELINE_FILENAME); - const baseline = existsSync(baselinePath) ? loadBaseline(cwd, baselinePath) : createBaseline([]); - const fingerprints = new Set(Object.keys(baseline.fingerprints)); + const baselineExists = existsSync(baselinePath); + const baseline = baselineExists ? loadBaseline(cwd, baselinePath) : createBaseline([]); + const issues = baselineExists ? filterIssues(result.issues, baseline) : [...result.issues]; const suppressions: string[] = []; + const processedIndexes = new Set(); const counts: TriageActionResult = { kept: 0, baselined: 0, suppressed: 0, skipped: 0 }; - const rl = createInterface({ - input: input.input ?? process.stdin, - output: input.output ?? process.stdout, - }); + const output = input.output ?? process.stdout; + const rl = input.ask + ? undefined + : createInterface({ + input: input.input ?? process.stdin, + output, + }); + const ask = input.ask ?? ((message: string) => rl!.question(message)); try { for (let index = 0; index < issues.length; index += 1) { + if (processedIndexes.has(index)) continue; const issue = issues[index]; if (!issue) continue; const rendered = formatIssue(issue, index + 1, issues.length); - process.stdout.write(`\n${rendered}\n`); - const rawAnswer = (await rl.question("Action [k]eep [b]aseline [s]uppress [n]ext [q]uit [B]atch rule: ")).trim(); + output.write(`\n${rendered}\n`); + const rawAnswer = (await ask("Action [k]eep [b]aseline [s]uppress [o]pen [n]ext [q]uit [B]atch rule: ")).trim(); const answer = rawAnswer.toLowerCase(); if (answer === "q" || answer === "quit") break; @@ -75,40 +83,59 @@ export async function runTriage(input: TriageInput): Promise counts.skipped += 1; continue; } + if (answer === "o" || answer === "open") { + output.write(`\n${formatIssueCreationSnippet(issue)}\n`); + counts.kept += 1; + continue; + } if (rawAnswer === "B" || answer === "batch" || answer === "b-rule") { - const batchAction = (await rl.question("Apply to all remaining findings of this rule with [k]eep [b]aseline [s]uppress? ")).trim().toLowerCase(); + const batchAction = (await ask("Apply to all remaining findings of this rule with [k]eep [b]aseline [s]uppress? ")).trim().toLowerCase(); + const suppressReason = isSuppressAction(batchAction) + ? await promptSuppressReason(ask, output) + : undefined; for (let cursor = index; cursor < issues.length; cursor += 1) { const candidate = issues[cursor]; if (!candidate || candidate.ruleId !== issue.ruleId) continue; applyTriageAction(candidate, batchAction, { dryRun: input.dryRun, baseline, - fingerprints, suppressions, counts, + suppressReason, + applySuppression: (directive) => applyInlineSuppression(cwd, options.target, candidate, directive, issues), }); + processedIndexes.add(cursor); } - break; + continue; } + const suppressReason = isSuppressAction(answer) + ? await promptSuppressReason(ask, output) + : undefined; applyTriageAction(issue, answer, { dryRun: input.dryRun, baseline, - fingerprints, suppressions, counts, + suppressReason, + applySuppression: (directive) => applyInlineSuppression(cwd, options.target, issue, directive, issues), }); } } finally { - rl.close(); + rl?.close(); } - if (!input.dryRun) { + if (!input.dryRun && counts.baselined > 0) { writeBaseline(cwd, baselinePath, baseline); + } + if (!input.dryRun) { if (suppressions.length > 0) { - process.stdout.write("\nSuggested suppression directives:\n"); - for (const directive of suppressions) process.stdout.write(directive); + output.write("\nApplied suppression directives:\n"); + for (const directive of suppressions) output.write(directive); } + } else if (suppressions.length > 0) { + output.write("\nSuggested suppression directives (dry run):\n"); + for (const directive of suppressions) output.write(directive); } return counts; @@ -120,27 +147,98 @@ function applyTriageAction( context: { dryRun?: boolean; baseline: ReturnType; - fingerprints: Set; suppressions: string[]; counts: TriageActionResult; + suppressReason?: string; + applySuppression: (directive: string) => void; }, ): void { - const fingerprint = issue.fingerprint ?? issue.id; if (action === "b" || action === "baseline") { - context.baseline.fingerprints[fingerprint] = (context.baseline.fingerprints[fingerprint] ?? 0) + 1; - context.fingerprints.add(fingerprint); + addIssuesToBaseline(context.baseline, [issue]); context.counts.baselined += 1; return; } if (action === "s" || action === "suppress") { - const reason = "triaged via debtlens triage"; - context.suppressions.push(runSuppress({ ruleId: issue.ruleId, reason })); + const reason = context.suppressReason?.trim(); + if (!reason) { + throw new Error("Suppression reason is required."); + } + const directive = runSuppress({ ruleId: issue.ruleId, reason }); + context.suppressions.push(directive); + if (!context.dryRun) context.applySuppression(directive); context.counts.suppressed += 1; return; } context.counts.kept += 1; } +function applyInlineSuppression( + cwd: string, + target: string, + issue: DebtIssue, + directive: string, + issues: DebtIssue[], +): void { + const line = issue.location?.startLine; + if (!line) throw new Error(`Cannot suppress ${issue.ruleId} in ${issue.file}: finding has no line location.`); + + const filePath = resolveIssueFile(cwd, target, issue.file); + const content = readFileSync(filePath, "utf8"); + const newline = content.includes("\r\n") ? "\r\n" : "\n"; + const lines = content.split(/\r?\n/); + const insertionIndex = line - 1; + const targetLine = lines[insertionIndex]; + if (targetLine === undefined) { + throw new Error(`Cannot suppress ${issue.ruleId} in ${issue.file}:${line}: line is outside the file.`); + } + + const indent = targetLine.match(/^\s*/)?.[0] ?? ""; + const comment = suppressionCommentForFile(filePath, directive.trim()); + lines.splice(insertionIndex, 0, `${indent}${comment}`); + writeFileSync(filePath, lines.join(newline), "utf8"); + + for (const candidate of issues) { + if (candidate.file !== issue.file || !candidate.location || candidate.location.startLine < line) continue; + candidate.location.startLine += 1; + if (candidate.location.endLine !== undefined) candidate.location.endLine += 1; + } +} + +function resolveIssueFile(cwd: string, target: string, issueFile: string): string { + const resolvedTarget = resolve(cwd, target); + if (existsSync(resolvedTarget) && statSync(resolvedTarget).isFile()) return resolvedTarget; + return resolve(resolvedTarget, issueFile); +} + +function suppressionCommentForFile(filePath: string, directive: string): string { + return /\.(?:py|rb)$/i.test(filePath) ? directive.replace(/^\/\//, "#") : directive; +} + +function formatIssueCreationSnippet(issue: DebtIssue): string { + const location = issue.location ? `${issue.file}:${issue.location.startLine}` : issue.file; + return [ + "Issue creation snippet:", + `Title: Address ${issue.ruleName} in ${issue.file}`, + `Body: DebtLens reported \`${issue.ruleId}\` at \`${location}\`. ${issue.message}`, + ...(issue.suggestion ? [`Suggested remediation: ${issue.suggestion}`] : []), + ].join("\n"); +} + +function isSuppressAction(action: string): boolean { + return action === "s" || action === "suppress"; +} + +async function promptSuppressReason( + ask: (message: string) => Promise, + output: NodeJS.WritableStream, +): Promise { + while (true) { + const reason = (await ask("Suppression reason (required): ")).trim(); + if (reason) return reason; + output.write("A reason is required for suppressions.\n"); + } +} + function formatIssue(issue: DebtIssue, index: number, total: number): string { const location = issue.location ? `${issue.file}:${issue.location.startLine}` : issue.file; return [ diff --git a/src/config/validateConfig.ts b/src/config/validateConfig.ts index b41cab7..a1687b7 100644 --- a/src/config/validateConfig.ts +++ b/src/config/validateConfig.ts @@ -1,5 +1,6 @@ import { isSeverity, severities } from "../core/severity.js"; import { gatePresets } from "../core/gatePresets.js"; +import { validateBudgetPattern } from "../core/budgets.js"; import { RULE_PACK_IDS } from "./packs.js"; import type { DebtLensConfig } from "../core/types.js"; import { DEBTLENS_PLUGIN_API_VERSION } from "../plugins/version.js"; @@ -304,6 +305,8 @@ function validateBudgets(errors: string[], value: unknown): void { return; } for (const [pattern, budget] of Object.entries(value)) { + const patternError = validateBudgetPattern(pattern); + if (patternError) errors.push(`budgets.${pattern} ${patternError}`); if (!isPlainObject(budget)) { errors.push(`budgets.${pattern} must be an object`); continue; diff --git a/src/core/baseline.ts b/src/core/baseline.ts index 565a6c0..639d83e 100644 --- a/src/core/baseline.ts +++ b/src/core/baseline.ts @@ -87,6 +87,22 @@ export function createBaseline(issues: DebtIssue[]): Baseline { }; } +/** Add findings to an existing baseline while keeping counts and snapshots in sync. */ +export function addIssuesToBaseline(baseline: Baseline, issues: DebtIssue[]): Baseline { + baseline.issues ??= {}; + for (const issue of issues) { + const fingerprint = computeFingerprint(issue); + const count = (baseline.fingerprints[fingerprint] ?? 0) + 1; + baseline.fingerprints[fingerprint] = count; + baseline.issues[fingerprint] = snapshotIssue(issue, count); + } + baseline.fingerprints = sortRecord(baseline.fingerprints); + baseline.issues = sortRecord(baseline.issues); + baseline.summary = summarizeBaselineFingerprints(baseline); + baseline.generatedAt = new Date().toISOString(); + return baseline; +} + export function writeBaseline(cwd: string, path: string, baseline: Baseline): string { const target = resolve(cwd, path); writeFileSync(target, `${JSON.stringify(baseline, null, 2)}\n`, "utf8"); diff --git a/src/core/budgets.ts b/src/core/budgets.ts index 92588fe..d9dd884 100644 --- a/src/core/budgets.ts +++ b/src/core/budgets.ts @@ -1,5 +1,6 @@ import { summarizeIssues } from "./issueAggregates.js"; import type { DebtIssue, ScanResult, Severity } from "./types.js"; +import micromatch from "micromatch"; export interface AreaBudget { maxIssues?: number; @@ -119,36 +120,22 @@ function normalizePath(file: string): string { return file.replaceAll("\\", "/"); } -function pathMatchesPattern(path: string, pattern: string): boolean { +export function validateBudgetPattern(pattern: string): string | undefined { const normalizedPattern = pattern.replaceAll("\\", "/"); - if (normalizedPattern.endsWith("/**")) { - const prefix = normalizedPattern.slice(0, -3); - return path === prefix || path.startsWith(`${prefix}/`); - } - if (normalizedPattern.includes("*")) { - let expression = ""; - for (let index = 0; index < normalizedPattern.length; index += 1) { - const char = normalizedPattern[index]; - const next = normalizedPattern[index + 1]; - if (char === "*" && next === "*") { - if (normalizedPattern[index + 2] === "/") { - expression += "(?:.*/)?"; - index += 2; - } else { - expression += ".*"; - index += 1; - } - } else if (char === "*") { - expression += "[^/]*"; - } else { - expression += escapeRegExp(char ?? ""); - } - } - return new RegExp(`^${expression}$`).test(path); + if (!normalizedPattern.trim()) return "must not be empty"; + if (normalizedPattern.startsWith("!")) return "must not use negation"; + try { + micromatch.makeRe(normalizedPattern, { nonegate: true }); + } catch { + return "must be a valid workspace glob"; } - return path === normalizedPattern || path.startsWith(`${normalizedPattern}/`); + return undefined; } -function escapeRegExp(value: string): string { - return value.replace(/[|\\{}()[\]^$+?.]/g, "\\$&"); +function pathMatchesPattern(path: string, pattern: string): boolean { + const normalizedPattern = pattern.replaceAll("\\", "/"); + const candidatePatterns = /[*?{}()[\]]/.test(normalizedPattern) + ? [normalizedPattern] + : [normalizedPattern, `${normalizedPattern}/**`]; + return candidatePatterns.some((candidate) => micromatch.isMatch(path, candidate, { dot: true, nonegate: true })); } diff --git a/src/core/calibrate.ts b/src/core/calibrate.ts index 0d4a1cd..89fccc5 100644 --- a/src/core/calibrate.ts +++ b/src/core/calibrate.ts @@ -1,4 +1,10 @@ -import { buildThresholdSuggestions, type ThresholdSuggestion } from "../cli/adoptionThresholds.js"; +import { + calibrationDiagnosticsForRules, + calibrationMetrics, + collectThresholdObservations, + type CalibrationDiagnostic, + type ThresholdSuggestion, +} from "../cli/adoptionThresholds.js"; import type { ScanOptions, ScanResult } from "../core/types.js"; export interface CalibrateOptions { @@ -8,6 +14,7 @@ export interface CalibrateOptions { export interface CalibrateResult { suggestions: ThresholdSuggestion[]; percentile: number; + diagnostics: CalibrationDiagnostic[]; } export function buildCalibrateSuggestions( @@ -15,41 +22,64 @@ export function buildCalibrateSuggestions( options: ScanOptions, calibrateOptions: CalibrateOptions, ): CalibrateResult { - const base = buildThresholdSuggestions(result, options); const percentile = clampPercentile(calibrateOptions.percentile); - const suggestions = base.map((suggestion) => { - const observed = percentileValue(suggestion.observedValues ?? [suggestion.observedP90], percentile / 100); - const suggested = Math.max(Math.ceil(observed * 1.05), Math.ceil(suggestion.current)); - return { - ...suggestion, - observedP90: observed, - suggested, - }; - }); - return { suggestions, percentile }; + const observations = collectThresholdObservations(result); + const suggestions = [...observations.entries()] + .map(([key, observedValues]) => { + const current = options.thresholds[key] ?? 0; + const observed = percentileValue(observedValues, percentile / 100); + return { + key, + current, + suggested: Math.max(1, Math.ceil(observed * 1.05)), + observedP90: observed, + samples: observedValues.length, + observedValues: [...observedValues], + }; + }) + .filter((suggestion) => suggestion.current > 0) + .sort((left, right) => left.key.localeCompare(right.key)); + const selected = options.rules ? new Set(options.rules) : undefined; + const unavailable = calibrationMetrics + .filter((entry) => (!selected || entry.ruleIds.some((ruleId) => selected.has(ruleId))) && !observations.has(entry.key)) + .map((entry) => ({ + key: entry.key, + ruleIds: entry.ruleIds, + reason: "no raw metric observations were emitted for the selected target", + })); + return { + suggestions, + percentile, + diagnostics: [...calibrationDiagnosticsForRules(options.rules), ...unavailable] + .sort((left, right) => left.key.localeCompare(right.key)), + }; } export function renderCalibrateReport(result: CalibrateResult): string { - if (result.suggestions.length === 0) { - return `No threshold suggestions at the p${result.percentile} percentile. Current defaults already match observed distributions.\n`; - } const lines = [ `DebtLens calibrate (p${result.percentile})`, "", - "Threshold".padEnd(34), - "Current", - "Suggested", - "Samples", - "-".repeat(34), - ...result.suggestions.map((suggestion) => - `${suggestion.key.padEnd(34)} ${String(suggestion.current).padEnd(7)} ${String(suggestion.suggested).padEnd(9)} ${suggestion.samples}`, - ), - "", - "Suggested config snippet:", - JSON.stringify({ - thresholds: Object.fromEntries(result.suggestions.map((suggestion) => [suggestion.key, suggestion.suggested])), - }, null, 2), ]; + if (result.suggestions.length > 0) { + lines.push( + "Threshold".padEnd(34), + "Current", + "Suggested", + "Samples", + "-".repeat(34), + ...result.suggestions.map((suggestion) => + `${suggestion.key.padEnd(34)} ${String(suggestion.current).padEnd(7)} ${String(suggestion.suggested).padEnd(9)} ${suggestion.samples}`, + ), + "", + "Suggested config snippet:", + JSON.stringify({ thresholds: Object.fromEntries(result.suggestions.map((suggestion) => [suggestion.key, suggestion.suggested])) }, null, 2), + ); + } else { + lines.push(`No numeric threshold suggestions at the p${result.percentile} percentile.`); + } + if (result.diagnostics.length > 0) { + lines.push("", "Not calibrated:", ...result.diagnostics.map((entry) => `- ${entry.key}: ${entry.reason}`)); + } return `${lines.join("\n")}\n`; } diff --git a/src/core/priority.ts b/src/core/priority.ts index c450cd3..da76003 100644 --- a/src/core/priority.ts +++ b/src/core/priority.ts @@ -62,7 +62,11 @@ export function sortIssuesByPayoff(issues: T[]): T[] { if (scoreDelta !== 0) return scoreDelta; const fileDelta = left.file.localeCompare(right.file); if (fileDelta !== 0) return fileDelta; - return (left.location?.startLine ?? 0) - (right.location?.startLine ?? 0); + const lineDelta = (left.location?.startLine ?? 0) - (right.location?.startLine ?? 0); + if (lineDelta !== 0) return lineDelta; + const ruleDelta = left.ruleId.localeCompare(right.ruleId); + if (ruleDelta !== 0) return ruleDelta; + return (left.fingerprint ?? left.id).localeCompare(right.fingerprint ?? right.id); }); } diff --git a/src/core/scanResultSchema.ts b/src/core/scanResultSchema.ts index ba795eb..0be5737 100644 --- a/src/core/scanResultSchema.ts +++ b/src/core/scanResultSchema.ts @@ -48,6 +48,20 @@ export function buildScanResultSchema(): Record { byRule: { type: "object", additionalProperties: { type: "integer", minimum: 0 } }, }, }; + const payoffTarget = { + type: "object", + additionalProperties: false, + required: ["id", "fingerprint", "ruleId", "file", "severity", "payoffScore"], + properties: { + id: { type: "string" }, + fingerprint: { type: "string" }, + ruleId: { type: "string" }, + file: { type: "string" }, + severity: severityValue, + payoffScore: { type: "number", minimum: 0 }, + location: issue.properties.location, + }, + }; const correlation = { type: "object", additionalProperties: false, @@ -296,6 +310,7 @@ export function buildScanResultSchema(): Record { filesScanned: { type: "integer", minimum: 0 }, rulesRun: { type: "integer", minimum: 0 }, elapsedMs: { type: "integer", minimum: 0 }, + topPayoffTargets: { type: "array", maxItems: 10, items: payoffTarget }, warnings: { type: "array", items: { type: "string" } }, filterStats: { type: "object", diff --git a/src/core/types.ts b/src/core/types.ts index 8d9d756..8fd4f52 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -37,6 +37,16 @@ export interface ReportedDebtIssue extends DebtIssue { fingerprint: string; } +export interface PayoffTarget { + id: string; + fingerprint: string; + ruleId: string; + file: string; + severity: Severity; + payoffScore: number; + location?: IssueLocation; +} + export interface SourceFileInfo { absolutePath: string; relativePath: string; @@ -541,6 +551,8 @@ export interface ScanSummary { profile?: ScanProfile; performance?: ScanPerformance; importGraph?: ImportGraph; + /** Deterministic, bounded payoff shortlist for machine consumers. */ + topPayoffTargets?: PayoffTarget[]; } export interface ScanResult { diff --git a/src/detectors/handlerDepth.ts b/src/detectors/handlerDepth.ts index ada9867..bb3b257 100644 --- a/src/detectors/handlerDepth.ts +++ b/src/detectors/handlerDepth.ts @@ -46,7 +46,7 @@ export const handlerDepthDetector: Detector = { evidence: [ `Control depth: ${depth} / ${maxDepth}`, `Nested callbacks: ${nestedCallbacks} / ${maxDepth}`, - ...(middlewareCount > 0 ? [`Middleware arguments: ${middlewareCount} / ${maxMiddleware}`] : []), + `Middleware arguments: ${middlewareCount} / ${maxMiddleware}`, ], suggestion: "Move validation, loading, and response branches into named middleware or service helpers so the handler reads as a flat request workflow.", })); diff --git a/src/detectors/longParameterList.ts b/src/detectors/longParameterList.ts index 22f89d2..bc1744d 100644 --- a/src/detectors/longParameterList.ts +++ b/src/detectors/longParameterList.ts @@ -66,7 +66,7 @@ function maybePushIssue( : `${name} has ${params.length} parameters.`, evidence: [ `Parameters: ${params.length} / ${maxParams}`, - ...(booleanCount > 0 ? [`Boolean parameters: ${booleanCount} / ${maxBooleans}`] : []), + `Boolean parameters: ${booleanCount} / ${maxBooleans}`, `Signature: ${truncateSignature(node)}`, ], suggestion: overBooleanBudget diff --git a/src/micromatch.d.ts b/src/micromatch.d.ts new file mode 100644 index 0000000..13c8055 --- /dev/null +++ b/src/micromatch.d.ts @@ -0,0 +1,14 @@ +declare module "micromatch" { + export interface Options { + dot?: boolean; + nonegate?: boolean; + } + + interface Micromatch { + isMatch(value: string, pattern: string, options?: Options): boolean; + makeRe(pattern: string, options?: Options): RegExp; + } + + const micromatch: Micromatch; + export default micromatch; +} diff --git a/tests/cli/calibrate.test.ts b/tests/cli/calibrate.test.ts new file mode 100644 index 0000000..2d1cc09 --- /dev/null +++ b/tests/cli/calibrate.test.ts @@ -0,0 +1,150 @@ +import assert from "node:assert/strict"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, dirname } from "node:path"; +import { spawnSync } from "node:child_process"; +import { createRequire } from "node:module"; +import { fileURLToPath } from "node:url"; +import { describe, it } from "node:test"; + +const repoRoot = join(dirname(fileURLToPath(import.meta.url)), "..", ".."); +const cliEntrypoint = join(repoRoot, "src", "cli", "index.ts"); +const localRequire = createRequire(import.meta.url); +const tsxLoader = localRequire.resolve("tsx"); + +const lowReactThresholds = [ + "--threshold", + "large-component.maxLines=50,large-component.maxHooks=3,large-component.maxBranches=5", +]; + +function runCli(args: string[], options: { cwd?: string } = {}) { + return spawnSync(process.execPath, ["--import", tsxLoader, cliEntrypoint, ...args], { + cwd: options.cwd ?? repoRoot, + encoding: "utf8", + }); +} + +describe("debtlens calibrate", () => { + it("prints percentile-based threshold suggestions", () => { + const result = runCli([ + "calibrate", + "examples/react", + "--pack", + "react", + ...lowReactThresholds, + "--percentile", + "90", + ]); + + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /DebtLens calibrate \(p90\)/); + assert.match(result.stdout, /large-component\.maxBranches/); + assert.match(result.stdout, /Suggested config snippet/); + }); + + it("collects below-threshold metrics and can recommend lower thresholds", () => { + const dir = mkdtempSync(join(tmpdir(), "debtlens-calibrate-distribution-")); + try { + const sourcePath = join(dir, "src", "Widgets.tsx"); + mkdirSync(join(dir, "src"), { recursive: true }); + writeFileSync(sourcePath, [ + "export function SmallWidget() {", + " const value = 1;", + " return
{value}
;", + "}", + "", + "export function MediumWidget() {", + " const first = 1;", + " const second = 2;", + " return
{first + second}
;", + "}", + "", + ].join("\n")); + writeFileSync(join(dir, "src", "helpers.ts"), [ + "export function addOne(value: number) {", + " return value + 1;", + "}", + "", + "export function addTwo(value: number) {", + " const next = value + 1;", + " return next + 1;", + "}", + "", + ].join("\n")); + + const result = runCli([ + "calibrate", + ".", + "--cwd", + dir, + "--rules", + "large-component,large-function", + "--percentile", + "50", + ]); + + assert.equal(result.status, 0, result.stderr); + const row = result.stdout.split("\n").find((line) => line.startsWith("large-component.maxLines")); + assert.ok(row); + const [, current, suggested, samples] = row.trim().split(/\s+/); + assert.equal(Number(current), 250); + assert.ok(Number(suggested) < Number(current)); + assert.equal(Number(samples), 2); + const functionRow = result.stdout.split("\n").find((line) => line.startsWith("large-function.maxLines")); + assert.ok(functionRow); + assert.equal(Number(functionRow.trim().split(/\s+/)[3]), 2); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("calibrates all numeric god-file axes and diagnoses the policy axis", () => { + const result = runCli(["calibrate", "examples/react", "--rules", "god-file", "--percentile", "90"]); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /god-file\.maxLines/); + assert.match(result.stdout, /god-file\.maxExports/); + assert.match(result.stdout, /god-file\.maxTopLevelDecls/); + assert.match(result.stdout, /god-file\.minAxes: minimum multi-axis trigger/); + assert.doesNotMatch(result.stdout, /Current defaults already match/); + }); + + it("merges suggested thresholds with --write", () => { + const dir = mkdtempSync(join(tmpdir(), "debtlens-calibrate-write-")); + try { + const configPath = join(dir, "debtlens.config.json"); + writeFileSync(configPath, JSON.stringify({ + pack: "react", + rules: ["large-component"], + minSeverity: "low", + thresholds: { + "large-component.maxLines": 500, + "large-component.maxHooks": 3, + "large-component.maxBranches": 5, + }, + })); + + const result = runCli([ + "calibrate", + "examples/react", + "--cwd", + repoRoot, + "--config", + configPath, + "--pack", + "react", + ...lowReactThresholds, + "--write", + ]); + assert.equal(result.status, 0, result.stderr); + + const config = JSON.parse(readFileSync(configPath, "utf8")) as { + minSeverity: string; + thresholds: Record; + }; + assert.equal(config.minSeverity, "low"); + assert.ok(config.thresholds["large-component.maxLines"] < 500); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/cli/scan.test.ts b/tests/cli/scan.test.ts index 4ab0ef5..5fc9eee 100644 --- a/tests/cli/scan.test.ts +++ b/tests/cli/scan.test.ts @@ -427,6 +427,44 @@ describe("debtlens scan output formats", () => { rmSync(dir, { recursive: true, force: true }); } }); + + it("sorts findings by payoff and prints top payoff targets", () => { + const result = runScan([ + "examples/react", + "--rules", + "todo-comment,prop-drilling", + "--sort", + "payoff", + "--format", + "json", + ]); + + assert.equal(result.status, 0); + const parsed = JSON.parse(result.stdout) as { + issues: Array<{ payoffScore?: number }>; + summary: { topPayoffTargets?: Array<{ payoffScore: number; fingerprint: string }> }; + }; + assert.ok(parsed.issues.length > 1); + assert.ok(parsed.issues.every((issue) => issue.payoffScore !== undefined)); + for (let index = 1; index < parsed.issues.length; index += 1) { + const previous = parsed.issues[index - 1]?.payoffScore ?? 0; + const current = parsed.issues[index]?.payoffScore ?? 0; + assert.ok(previous >= current); + } + assert.ok(parsed.summary.topPayoffTargets); + assert.ok((parsed.summary.topPayoffTargets?.length ?? 0) <= 10); + assert.deepEqual(parsed.summary.topPayoffTargets?.map((target) => target.payoffScore), parsed.issues.slice(0, 10).map((issue) => issue.payoffScore)); + + const terminal = runScan([ + "examples/react", + "--rules", + "todo-comment", + "--sort", + "payoff", + ]); + assert.equal(terminal.status, 0); + assert.match(terminal.stdout, /Top payoff targets/); + }); }); describe("debtlens scan fail-on confidence", () => { diff --git a/tests/cli/triage.test.ts b/tests/cli/triage.test.ts index 7e96225..ead03a4 100644 --- a/tests/cli/triage.test.ts +++ b/tests/cli/triage.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync, readFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { Readable, Writable } from "node:stream"; @@ -27,4 +27,208 @@ describe("debtlens triage", () => { rmSync(dir, { recursive: true, force: true }); } }); + + for (const [label, answers, expected] of [ + ["quit", ["q"], { kept: 0, baselined: 0, suppressed: 0, skipped: 0 }], + ["keep", ["k"], { kept: 1, baselined: 0, suppressed: 0, skipped: 0 }], + ] as const) { + it(`does not create a baseline file after a non-dry-run ${label} flow`, async () => { + const dir = mkdtempSync(join(tmpdir(), `debtlens-triage-${label}-`)); + try { + mkdirSync(join(dir, "src")); + writeFileSync(join(dir, "src", "app.ts"), "// TODO triage me\nexport const value = 1;\n"); + const baselinePath = join(dir, "debtlens-baseline.json"); + const queue = [...answers]; + const counts = await runTriage({ + target: ".", + cwd: dir, + baselinePath, + cliOptions: { rules: "todo-comment" }, + output: new Writable({ write(_chunk, _encoding, callback) { callback(); } }), + ask: async () => queue.shift() ?? "q", + }); + + assert.deepEqual(counts, expected); + assert.equal(existsSync(baselinePath), false); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + } + + it("baselines a finding when requested", async () => { + const dir = mkdtempSync(join(tmpdir(), "debtlens-triage-baseline-")); + try { + mkdirSync(join(dir, "src")); + writeFileSync(join(dir, "src", "app.ts"), "// TODO triage me\nexport const value = 1;\n"); + const baselinePath = join(dir, "debtlens-baseline.json"); + + const counts = await runTriage({ + target: ".", + cwd: dir, + baselinePath, + cliOptions: { rules: "todo-comment" }, + input: Readable.from(["b\n", "q\n"]), + output: new Writable({ write(_chunk, _encoding, callback) { callback(); } }), + }); + + assert.deepEqual(counts, { kept: 0, baselined: 1, suppressed: 0, skipped: 0 }); + const baseline = JSON.parse(readFileSync(baselinePath, "utf8")) as { + fingerprints: Record; + summary: { totalIssues: number; byRule: Record }; + issues: Record; + }; + assert.equal(Object.keys(baseline.fingerprints).length, 1); + assert.equal(baseline.summary.totalIssues, 1); + assert.equal(baseline.summary.byRule["todo-comment"], 1); + assert.equal(Object.values(baseline.issues)[0]?.ruleId, "todo-comment"); + assert.equal(Object.values(baseline.issues)[0]?.count, 1); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("does not re-present or inflate a finding already in the loaded baseline", async () => { + const dir = mkdtempSync(join(tmpdir(), "debtlens-triage-rerun-")); + try { + mkdirSync(join(dir, "src")); + writeFileSync(join(dir, "src", "app.ts"), "// TODO triage me\nexport const value = 1;\n"); + const baselinePath = join(dir, "debtlens-baseline.json"); + await runTriage({ + target: ".", + cwd: dir, + baselinePath, + cliOptions: { rules: "todo-comment" }, + output: new Writable({ write(_chunk, _encoding, callback) { callback(); } }), + ask: async () => "b", + }); + const before = readFileSync(baselinePath, "utf8"); + let prompts = 0; + const output: string[] = []; + + const counts = await runTriage({ + target: ".", + cwd: dir, + baselinePath, + cliOptions: { rules: "todo-comment" }, + output: new Writable({ write(chunk, _encoding, callback) { output.push(String(chunk)); callback(); } }), + ask: async () => { prompts += 1; return "b"; }, + }); + + assert.deepEqual(counts, { kept: 0, baselined: 0, suppressed: 0, skipped: 0 }); + assert.equal(prompts, 0); + assert.equal(output.join(""), ""); + assert.equal(readFileSync(baselinePath, "utf8"), before); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("prompts for a suppression reason", async () => { + const dir = mkdtempSync(join(tmpdir(), "debtlens-triage-suppress-")); + try { + mkdirSync(join(dir, "src")); + writeFileSync(join(dir, "src", "app.ts"), "// TODO triage me\nexport const value = 1;\n"); + const output: string[] = []; + const answers = ["s", "tracked in PROJ-42"]; + const out = new Writable({ + write(chunk, _encoding, callback) { + output.push(String(chunk)); + callback(); + }, + }); + + const counts = await runTriage({ + target: ".", + cwd: dir, + dryRun: true, + cliOptions: { rules: "todo-comment" }, + output: out, + ask: async () => answers.shift() ?? "", + }); + + assert.deepEqual(counts, { kept: 0, baselined: 0, suppressed: 1, skipped: 0 }); + assert.match(output.join(""), /debtlens-disable-next-line todo-comment -- tracked in PROJ-42/); + assert.doesNotMatch(readFileSync(join(dir, "src", "app.ts"), "utf8"), /debtlens-disable-next-line/); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("writes an inline suppression that hides the finding", async () => { + const dir = mkdtempSync(join(tmpdir(), "debtlens-triage-write-suppress-")); + try { + mkdirSync(join(dir, "src")); + const sourcePath = join(dir, "src", "app.ts"); + writeFileSync(sourcePath, "// TODO triage me\nexport const value = 1;\n"); + const answers = ["s", "tracked in PROJ-42"]; + + const counts = await runTriage({ + target: ".", + cwd: dir, + cliOptions: { rules: "todo-comment" }, + output: new Writable({ write(_chunk, _encoding, callback) { callback(); } }), + ask: async () => answers.shift() ?? "q", + }); + + assert.equal(counts.suppressed, 1); + assert.equal(existsSync(join(dir, "debtlens-baseline.json")), false); + assert.match(readFileSync(sourcePath, "utf8"), /^\/\/ debtlens-disable-next-line todo-comment -- tracked in PROJ-42\n\/\/ TODO/); + + const verificationAnswers = ["q"]; + const verificationOutput: string[] = []; + const verified = await runTriage({ + target: ".", + cwd: dir, + dryRun: true, + cliOptions: { rules: "todo-comment" }, + output: new Writable({ + write(chunk, _encoding, callback) { + verificationOutput.push(String(chunk)); + callback(); + }, + }), + ask: async () => verificationAnswers.shift() ?? "q", + }); + assert.deepEqual(verified, { kept: 0, baselined: 0, suppressed: 0, skipped: 0 }); + assert.equal(verificationOutput.join(""), ""); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("prints an issue snippet and continues after a batch-by-rule action", async () => { + const dir = mkdtempSync(join(tmpdir(), "debtlens-triage-batch-")); + try { + mkdirSync(join(dir, "src")); + writeFileSync(join(dir, "src", "app.ts"), [ + "// TODO first", + "// TODO second", + "try { run(); } catch (error) {}", + "", + ].join("\n")); + const output: string[] = []; + const answers = ["B", "k", "o"]; + + const counts = await runTriage({ + target: ".", + cwd: dir, + dryRun: true, + cliOptions: { rules: "todo-comment,empty-catch" }, + output: new Writable({ + write(chunk, _encoding, callback) { + output.push(String(chunk)); + callback(); + }, + }), + ask: async () => answers.shift() ?? "q", + }); + + assert.deepEqual(counts, { kept: 2, baselined: 0, suppressed: 0, skipped: 0 }); + assert.match(output.join(""), /Issue creation snippet:/); + assert.match(output.join(""), /empty-catch/); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); }); diff --git a/tests/config/validateConfig.test.ts b/tests/config/validateConfig.test.ts index 28c0290..1d261d2 100644 --- a/tests/config/validateConfig.test.ts +++ b/tests/config/validateConfig.test.ts @@ -104,6 +104,12 @@ describe("validateConfigShape", () => { assert.deepEqual(result.errors, []); }); + it("rejects negated budget globs instead of silently changing their meaning", () => { + const result = validateConfigShape({ budgets: { "!src/generated/**": { maxIssues: 0 } } }); + assert.equal(result.valid, false); + assert.match(result.errors.join("\n"), /must not use negation/); + }); + it("rejects invalid budgets, badge, and payoff priority config", () => { const result = validateConfigShape({ budgets: { diff --git a/tests/core/baseline.test.ts b/tests/core/baseline.test.ts index 027a10c..b6bc82b 100644 --- a/tests/core/baseline.test.ts +++ b/tests/core/baseline.test.ts @@ -2,6 +2,7 @@ import assert from "node:assert/strict"; import { describe, it } from "node:test"; import { applyBaseline, + addIssuesToBaseline, compareBaseline, compareBaselineDetailed, computeFingerprint, @@ -132,6 +133,17 @@ describe("compareBaseline", () => { }); describe("baseline maintenance helpers", () => { + it("adds triaged findings with summary and snapshot metadata kept consistent", () => { + const finding = issue({ ruleId: "todo-comment", severity: "low" }); + const baseline = createBaseline([]); + addIssuesToBaseline(baseline, [finding]); + + assert.equal(baseline.summary?.totalIssues, 1); + assert.equal(baseline.summary?.byRule["todo-comment"], 1); + assert.equal(baseline.summary?.bySeverity.low, 1); + assert.equal(baseline.issues?.[finding.fingerprint]?.ruleId, "todo-comment"); + assert.equal(compareBaselineDetailed([finding], baseline).newIssues.length, 0); + }); it("reports detailed new, resolved, stale, and changed fingerprint data with occurrence counts", () => { const repeated = issue({ fingerprint: "dl_repeated" }); const fresh = issue({ diff --git a/tests/core/budgets.test.ts b/tests/core/budgets.test.ts index 5c21b67..ea322d7 100644 --- a/tests/core/budgets.test.ts +++ b/tests/core/budgets.test.ts @@ -59,6 +59,16 @@ describe("budget evaluation", () => { assert.equal(evaluation?.areas[0]?.issueCount, 2); }); + it("uses workspace-compatible brace and character-class globs", () => { + const result = makeResult([ + { id: "1", fingerprint: "1", ruleId: "todo-comment", ruleName: "Todo", severity: "high", confidence: 1, file: "packages/api/src/a.ts", message: "todo", tags: [] }, + { id: "2", fingerprint: "2", ruleId: "todo-comment", ruleName: "Todo", severity: "low", confidence: 1, file: "packages/docs/src/a.ts", message: "todo", tags: [] }, + ]); + const evaluation = evaluateBudgets(result, { "packages/{api,web}/src/[a-z].ts": { maxHigh: 0 } }); + assert.equal(evaluation?.areas[0]?.issueCount, 1); + assert.equal(evaluation?.breached, true); + }); + it("renders a budget report table", () => { const result = makeResult([]); const evaluation = evaluateBudgets(result, { diff --git a/tests/core/calibrationMetadata.test.ts b/tests/core/calibrationMetadata.test.ts new file mode 100644 index 0000000..0cdfdca --- /dev/null +++ b/tests/core/calibrationMetadata.test.ts @@ -0,0 +1,27 @@ +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { describe, it } from "node:test"; +import fg from "fast-glob"; +import { calibrationDiagnostics, calibrationMetrics } from "../../src/cli/adoptionThresholds.js"; + +describe("calibration metadata", () => { + it("classifies every literal threshold used by a built-in detector", () => { + const used = new Set(); + for (const file of fg.sync("src/detectors/**/*.ts")) { + const source = readFileSync(file, "utf8"); + for (const match of source.matchAll(/getThreshold\("([^"]+)"/g)) { + if (match[1]) used.add(match[1]); + } + } + const classified = new Set([ + ...calibrationMetrics.map((entry) => entry.key), + ...calibrationDiagnostics.map((entry) => entry.key), + ]); + assert.deepEqual([...used].filter((key) => !classified.has(key)).sort(), []); + }); + + it("does not claim the same threshold is both calibrated and non-calibratable", () => { + const supported = new Set(calibrationMetrics.map((entry) => entry.key)); + assert.deepEqual(calibrationDiagnostics.filter((entry) => supported.has(entry.key)).map((entry) => entry.key), []); + }); +}); diff --git a/tests/core/priority.test.ts b/tests/core/priority.test.ts index 4350e18..dcd01d2 100644 --- a/tests/core/priority.test.ts +++ b/tests/core/priority.test.ts @@ -1,6 +1,6 @@ import assert from "node:assert/strict"; import { describe, it } from "node:test"; -import { computePayoffScore, sortIssuesByPayoff } from "../../src/core/priority.js"; +import { computePayoffScore, sortIssuesByPayoff, topPayoffIssues } from "../../src/core/priority.js"; import type { DebtIssue } from "../../src/core/types.js"; const baseIssue: DebtIssue = { @@ -31,4 +31,20 @@ describe("payoff ranking", () => { const sorted = sortIssuesByPayoff([right, left]); assert.equal(sorted[0]?.id, "left"); }); + + it("uses rule and identity tie-breakers for otherwise identical targets", () => { + const first: DebtIssue = { ...baseIssue, id: "a", ruleId: "a-rule", payoffScore: 5 }; + const second: DebtIssue = { ...baseIssue, id: "b", ruleId: "b-rule", payoffScore: 5 }; + assert.deepEqual(sortIssuesByPayoff([second, first]).map((issue) => issue.id), ["a", "b"]); + }); + + it("bounds machine-facing payoff targets to ten by default", () => { + const issues = Array.from({ length: 12 }, (_, index) => ({ + ...baseIssue, + id: String(index), + payoffScore: index, + })); + assert.equal(topPayoffIssues(issues).length, 10); + assert.equal(topPayoffIssues(issues)[0]?.payoffScore, 11); + }); }); diff --git a/tests/core/scanResultSchema.test.ts b/tests/core/scanResultSchema.test.ts index 4965d72..31641c5 100644 --- a/tests/core/scanResultSchema.test.ts +++ b/tests/core/scanResultSchema.test.ts @@ -23,6 +23,7 @@ describe("ScanResult JSON schema", () => { deltaFromBaseline?: { required: string[] }; correlations?: { items: { required: string[] } }; duplicateClusters?: { items: { required: string[] } }; + topPayoffTargets?: { maxItems: number; items: { required: string[] } }; importGraph?: { required: string[]; properties: { edges: { items: { required: string[] } } } }; hotspots?: { required: string[]; @@ -53,6 +54,8 @@ describe("ScanResult JSON schema", () => { assert.ok(schema.properties.summary.properties.deltaFromBaseline?.required.includes("totalDelta")); assert.ok(schema.properties.summary.properties.correlations?.items.required.includes("rules")); assert.ok(schema.properties.summary.properties.duplicateClusters?.items.required.includes("locations")); + assert.equal(schema.properties.summary.properties.topPayoffTargets?.maxItems, 10); + assert.ok(schema.properties.summary.properties.topPayoffTargets?.items.required.includes("payoffScore")); assert.ok(schema.properties.summary.properties.importGraph?.required.includes("edges")); assert.ok(schema.properties.summary.properties.importGraph?.properties.edges.items.required.includes("inCycle")); assert.ok(schema.properties.summary.properties.hotspots?.required.includes("ranking"));