diff --git a/docs/specs/project-development-analytics.md b/docs/specs/project-development-analytics.md new file mode 100644 index 0000000..98d6447 --- /dev/null +++ b/docs/specs/project-development-analytics.md @@ -0,0 +1,51 @@ +# Spec: project development analytics on the dashboard + +Builtin-format mirror of the OpenSpec change +[`project-development-analytics`](../../openspec/changes/project-development-analytics/). The change's +`proposal.md` and `specs/project-analytics/spec.md` carry the full requirements and scenarios; this file +satisfies the repo's builtin spec gate and states the problem, requirements, and acceptance in brief. + +## Problem + +The governance dashboard shows the pipeline (PRs bucketed by stage) but not whether the project is getting +healthier, what the work is about against the project's goal and invariants, or whether the agent review is +actually keeping defects out. A project can merge steadily and still drift: all features and no tests, all +code and no docs, velocity up and correctness down. There is no single view of where the project sits today +and where it heads if the open work merges, and no quantifiable read on building bug-free software. + +## Requirements + +1. A two-state summary: submitted PRs as the to-be state, merged PRs as the current-state. +2. A read of what the work is about (leading topics per state) against the declared project goal and + invariants. +3. A six-dimension radar balance map (0 centre to 5 point) overlaying current-state (merged) and to-be + (merged plus on-track open PRs); a changes-requested PR is not projected as if it will merge unchanged. +4. Six dimensions: Correctness, Verification, Documentation, Spec conformance, Governance integrity, Flow, + each scored 0 to 5, with non-applicable PRs excluded from a dimension's denominator. +5. Governance KPI ratios (spec-fit, review pass/fail, tests, docs, review categories, MUST-invariant + conform/violate, core-change attempts, malicious), each as total, percentage, and failing count. +6. The escaped-defect KPI framed as the review agent's false-negative rate, linked by an `Escaped-from: #N` + trailer on a bug-lane PR or a bug-labelled issue, feeding Correctness. +7. A temporal trend (per-window series) for the headline signals, and submitted-versus-merged throughput. +8. Minimum-signal handling: below a metric's threshold the dashboard shows a short "still filling in" note, + not a misleading value; an existing repo scores from its history; a metric reveals itself as signal + accrues. "Not enough signal" wins over a maximum score at low counts. +9. Provenance and privacy unchanged: only facts already on GitHub, private stays behind auth, `--public` + stays the explicit opt-in. + +Two producer changes this depends on: intake exposes the spec result (`spec_ok`) as its own signal so the +spec-conformance metric is not read from the rolled-up intake verdict; and the review emits a +machine-readable per-lens result so categories, malicious, and impact-core are queryable without scraping +the rendered comment. + +## Acceptance criteria + +- Both states render, including an empty to-be state, without error. +- Each dimension has a current-state and a to-be score in [0, 5] or a "not enough signal" state; Correctness + does not read 5 on a project with too few green merges. +- Every ratio and dimension is computed over an explicit, stated window whose denominator covers the whole + window (paginated), never an unstated recent slice. +- The escaped-defect count links from a bug PR or issue, feeds Correctness, and is presented as the review + false-negative rate. +- A below-threshold metric shows a note rather than a number, and `--public` adds no new disclosure. +- A worked HTML example was produced as a design reference for the view. diff --git a/openspec/changes/project-development-analytics/.openspec.yaml b/openspec/changes/project-development-analytics/.openspec.yaml new file mode 100644 index 0000000..5e6d53a --- /dev/null +++ b/openspec/changes/project-development-analytics/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-24 diff --git a/openspec/changes/project-development-analytics/proposal.md b/openspec/changes/project-development-analytics/proposal.md new file mode 100644 index 0000000..2a4ea15 --- /dev/null +++ b/openspec/changes/project-development-analytics/proposal.md @@ -0,0 +1,89 @@ +## Why + +The governance dashboard shows the pipeline: pull requests bucketed by stage (awaiting review, in +progress, changes requested, merged), by lane. That answers "what is moving through the gate right now." +It does not answer the questions a project owner actually asks: + +- **What is the work about?** Across everything submitted and everything merged, what are the changes + mostly doing, and does that match the project's stated goal and its invariants? +- **Is the project healthy and balanced, or lopsided?** A project can merge steadily and still drift: + all features, no tests; all code, no docs; velocity up, correctness down. There is no single view of + where the project sits today and where it is heading if the open work merges. +- **Are we building bug-free software?** The point of the gate is fewer escaped defects. Nothing today + counts the failures that matter most: a change that passed review green, passed its tests, and still + turned out to carry a bug. + +GitHub already holds every raw fact (each PR, each check, each merge). The dashboard's job is not to +repeat GitHub; it is to **synthesise** those facts into a small number of quantifiable signals a human +can read at a glance, and to make the trajectory visible: current-state (what has merged) versus the +to-be state (what merges if the open PRs land). + +## What Changes + +Add a **project-analytics** section to the dashboard, above or beside the existing stage buckets. It is +derived entirely from PR facts the dashboard already fetches, plus a small set of declared link +conventions for the signals GitHub does not carry directly. Nothing here changes the gate; it is a +read-only view. + +1. **Two-state summary.** A count and short digest of every **submitted** PR (open = the *to-be* state) + and every **merged** PR (the *current-state*), so the two are always read side by side. + +2. **What the work is about.** The most common topics across submitted PRs and across merged PRs (by + lane, declared scope, and changed-area), and a plain-language read of what those topics mean against + the project's **invariants** and its **stated goal** (both declared in config). + +3. **A balance map (radar / spider chart).** A star-shaped chart with **6 fixed dimensions**, each scored + `0` at the centre to `5` at the point, drawn as **two overlaid polygons**: the current-state (from + merged PRs) and the to-be state (from merged plus open PRs, projecting the open work as if merged). The + shape shows at a glance whether the project is balanced across the principles of a healthy software + project or spiking in one direction, the way an attribute chart does for sports equipment or a game + character. The six dimensions and their scoring are defined in the spec. + +4. **Governance KPI ratios.** Totals and percentages that quantify how the process is going: how many + incoming PRs fit the spec/OpenSpec standard; how many pass review versus not (with the failing count + called out); tests created and tests passing and doc updates from merged PRs and their trend; which + review categories findings fall into; how many PRs conform to versus violate a MUST invariant; how many + attempted to change the core business logic of the code; and how many were flagged malicious. + +5. **Bug-free-software KPIs.** The headline performance numbers: submitted versus merged, and the + **escaped-defect count** framed as the **review agent's false-negative rate** - a defect reported + against a PR that had passed review green and passed its tests yet still shipped. Green-then-buggy means + the review missed it, so for a project that runs the agent review this is the sharpest read of whether + the agent fleet is actually catching bugs. It is the number the whole gate exists to drive down. + +6. **A temporal trend.** The radar answers current-vs-to-be; alongside it, a per-window series (a small + trend line) for the headline signals, so an owner sees whether escaped defects, throughput, and spec-fit + are rising or falling, not only their latest value. + +A worked HTML example of the whole view (radar, KPIs, escaped-defect hero, sparklines, and the +still-filling-in note) was produced as a design reference for this change. + +## What this depends on + +Two of the KPIs cannot come from what the pipeline emits today, so this change carries the small producer +changes they need: + +- **Spec conformance is its own signal.** Intake rolls the spec check together with disclosure, DCO, and + lane, and a chore lane is spec-exempt, so "passed intake" is not "spec-conformant." Intake must expose + the spec result separately. +- **The review emits a machine-readable per-lens result.** Review categories, malicious flags, and + impact-core findings already exist inside the review, but only the rendered comment is queryable. A + structured per-lens emission (a status per lens, or a parseable block) unlocks all three at once, instead + of the dashboard scraping a comment. The MUST-invariant KPI stays "not enough signal" until a review + emits an explicit invariant check; the spec marks it blocked on that producer rather than implying it is + derivable. + +## Impact + +- Affected specs: new capability `project-analytics`. +- Affected code (at build time, not in this change): `cli/dashboard.py` (a new analytics model + render + section, plus extending `fetch` to read merged-PR statuses and files and to paginate the declared + window), its HTML, and `cli/dashboard.test.sh`; `.github/asdd/intake-check.sh` (expose `spec_ok` + separately); the review path (`post-review.sh` or a per-lens status) for the machine-readable emission. +- New declared conventions (config + trailers) for the signals GitHub does not carry: the project goal and + invariants, and the escaped-defect link (`Escaped-from: #N` on a bug-lane PR or a bug-labelled issue). + Every KPI degrades to "not enough signal" rather than a wrong number when its input is absent, and each + metric declares a minimum signal so a new project shows a note while an existing repo scores from history. +- Privacy and provenance are unchanged from the dashboard's existing stance: every rendered fact is already + visible on GitHub, private deployments stay behind auth, and `--public` remains the explicit opt-in for + publication. diff --git a/openspec/changes/project-development-analytics/specs/project-analytics/spec.md b/openspec/changes/project-development-analytics/specs/project-analytics/spec.md new file mode 100644 index 0000000..d072178 --- /dev/null +++ b/openspec/changes/project-development-analytics/specs/project-analytics/spec.md @@ -0,0 +1,205 @@ +## ADDED Requirements + +### Requirement: Two-state PR summary +The dashboard MUST render a summary of every submitted (open) pull request and every merged pull request, +labelled so the two are read as distinct states: **submitted = the to-be state** (what the project becomes +if the open work merges) and **merged = the current-state** (what the project is now). Each state MUST show +a total count and a short per-PR digest (number, title, lane, declared scope). + +#### Scenario: Both states are shown side by side +- **WHEN** the dashboard renders for a repository with open and merged PRs +- **THEN** it shows a submitted (to-be) group and a merged (current-state) group, each with its total and a + per-PR digest +- **AND** a repository with no open PRs still shows the merged (current-state) group and an empty to-be + group, not an error + +### Requirement: Analytics window and merged-PR signals +The analytics MUST score the current-state from merged PRs, which means it MUST read each merged PR's check +statuses and changed files, not only open PRs. Every ratio and dimension MUST be computed over an explicit, +stated **window** (for example a PR count, a date range, or all history), and MUST NOT silently measure over +only the most-recently-updated page of PRs. When the repository has more PRs than one fetch page, the +analytics MUST paginate to cover the declared window rather than report a partial denominator as if it were +the whole. + +#### Scenario: Current-state reads merged-PR signals +- **WHEN** the analytics scores the current-state +- **THEN** it uses the check statuses and changed files of merged PRs, not only open PRs + +#### Scenario: The measured window is explicit and complete +- **WHEN** the dashboard renders a ratio or dimension +- **THEN** the window it was computed over is stated on the page +- **AND** the denominator covers that whole window, paginating when the repository exceeds one fetch page, + so a large repository is never scored over an unstated recent slice + +### Requirement: What the work is about, against goal and invariants +The dashboard MUST report the most common topics across submitted PRs and, separately, across merged PRs, +derived from each PR's lane, declared scope, and changed areas. It MUST present those topics against the +project's declared **goal** and **invariants** (read from config), stating in plain language whether the +weight of the work supports the goal or diverges from it. When no goal or invariants are declared, it MUST +say so rather than invent a judgement. + +#### Scenario: Topics summarised for each state +- **WHEN** submitted and merged PRs carry lanes, declared scopes, and changed paths +- **THEN** the dashboard lists the leading topics for the to-be state and for the current-state separately + +#### Scenario: Topics read against the declared goal +- **WHEN** a project goal and invariants are declared in config +- **THEN** the dashboard states whether the leading topics support or diverge from that goal and names any + topic that touches an invariant +- **WHEN** no goal or invariants are declared +- **THEN** the dashboard reports the topics and states that none are declared to judge them against + +### Requirement: Balance map across six fixed dimensions +The dashboard MUST render a radar (spider) chart with exactly **six fixed dimensions**, each on a scale from +`0` at the centre to `5` at the outer point, drawn as **two overlaid polygons**: the current-state scored +from merged PRs, and the to-be state scored from merged plus open PRs. An open PR whose review is requesting +changes MUST NOT be projected into the to-be state as if it will merge unchanged; the to-be projection MUST +exclude or down-weight a changes-requested PR, so it reflects the work that is actually on track to land. + +#### Scenario: Two polygons on one chart +- **WHEN** the dashboard renders analytics for a repository +- **THEN** the radar shows a current-state polygon and a to-be polygon over the same six labelled axes, each + axis scaled 0 at centre to 5 at the point + +#### Scenario: To-be excludes work the review is blocking +- **WHEN** an open PR's review recommendation is request-changes +- **THEN** that PR is excluded or down-weighted in the to-be projection, so the to-be polygon is not inflated + by work that is not on track to merge + +#### Scenario: A missing dimension does not break the chart +- **WHEN** a dimension has no signal to score +- **THEN** that axis renders with a "not enough signal" marker rather than a misleading value, and the rest of + the chart still renders + +### Requirement: Definition of the six balance dimensions +The six dimensions MUST be the principles below, each scored `0` to `5` from PR-derived signals, for the +current-state (over merged PRs) and the to-be state (over merged plus on-track open PRs) by the same rule. + +1. **Correctness** - the share of green-merged PRs (passed review, passed tests) that did NOT later produce a + reported escaped defect. +2. **Verification** - among **code-changing** PRs only, the share that added or extended tests AND passed the + test gate. A docs-only or non-code PR MUST be excluded from this denominator, not scored zero. +3. **Documentation** - among **user-facing** PRs only, the share that updated docs or the impact log. +4. **Spec conformance** - the share of PRs that met the spec / OpenSpec readiness standard at intake, counting + only the spec signal (see the spec-conformance requirement), over lanes that are not spec-exempt. +5. **Governance integrity** - the share of merges that passed the agent and human review with no override and + no MUST-invariant violation, reduced by any malicious or against-invariant attempts. +6. **Flow** - submitted-to-merged conversion and the absence of long-stuck PRs. + +A dimension MUST NOT report a confident score below its minimum-signal threshold (see the minimum-signal +requirement). In particular Correctness MUST NOT default to `5` on a project with too few green merges to +judge: "not enough signal" MUST win over the maximum score at low counts, so an unproven project never reads +as flawless. + +#### Scenario: Each dimension is scored for both states by one rule +- **WHEN** the analytics model runs +- **THEN** each of the six dimensions has a current-state score and a to-be score in `[0, 5]` produced by the + same scoring rule over the respective PR set, or a "not enough signal" state + +#### Scenario: Non-applicable PRs leave a denominator +- **WHEN** the Verification or Documentation dimension is scored +- **THEN** PRs the dimension does not apply to (a docs-only PR for Verification, a non-user-facing PR for + Documentation) are excluded from the denominator rather than counted as failures + +### Requirement: Spec conformance uses its own signal, not the intake verdict +The spec-conformance metric MUST NOT be read from the single intake pass/fail, because intake rolls the spec +check together with disclosure, DCO, and lane, and a spec-exempt lane (chore) passes intake with no spec. The +intake gate MUST expose the spec result as its own signal, and the metric MUST exclude spec-exempt lanes from +its denominator, so "passed intake" is never miscounted as "spec-conformant." + +#### Scenario: A chore PR is not counted as spec-conformant +- **WHEN** a chore-lane PR passes intake without a spec (spec-exempt) +- **THEN** it is excluded from the spec-conformance denominator, not counted as a conforming spec + +### Requirement: Machine-readable per-lens review output +So the dashboard can report review categories, malicious flags, and impact-core findings structurally rather +than by scraping a rendered comment, the review MUST emit a machine-readable per-lens result (a per-lens +status context, or a structured block the dashboard parses), carrying each lens name, its verdict, and its +findings. The human-facing comment is unchanged; this is an additional structured emission. + +#### Scenario: Per-lens results are queryable +- **WHEN** a review completes +- **THEN** the dashboard can read each lens (code, security, spec, quality, impact) with its verdict and + findings from a structured source, without parsing the prose comment + +### Requirement: Governance KPI ratios +The dashboard MUST render the following as totals and percentages, with the count that fails or violates +always shown alongside the percentage: incoming PRs that fit the spec standard versus not; PRs that pass +review versus not (the failing count called out); tests created, tests passing, and doc updates from merged +PRs, each with an up or down trend; the review categories findings fall into, with counts; PRs that conform +to versus violate a MUST invariant; PRs that attempted to change the core business logic; and PRs flagged as +malicious. A PR is **malicious** when the security lens returns a blocking or injected-instruction verdict +(derived from the per-lens output, not from a label that nothing applies). A PR is a **core-change** when it +touches a declared `protected_path` or the impact lens marks it normative-core. A MUST-invariant +conform/violate count MUST come from an explicit review-emitted invariant check; until a producer emits that, +this KPI MUST read "not enough signal" rather than imply a derived number. + +#### Scenario: Ratios show total, percentage, and the failing count +- **WHEN** the dashboard renders the KPI ratios +- **THEN** each ratio shows a total, a percentage, and the absolute number of the failing or violating side + +#### Scenario: Malicious is derived from the security lens, not a label +- **WHEN** the security lens returns a blocking or injected-instruction verdict on a PR +- **THEN** that PR is counted as malicious, without depending on any label that the pipeline does not apply + +#### Scenario: An invariant KPI with no producer reads as unavailable +- **WHEN** no review-emitted invariant check exists +- **THEN** the MUST-invariant conform/violate KPI reads "not enough signal", not a fabricated `100%` + +### Requirement: Escaped-defect KPI as the review false-negative signal +The dashboard MUST count **escaped defects**: a defect reported against a PR that had passed review green and +passed its tests, yet still shipped the defect. The link MUST come from a declared convention: a bug-lane PR +**or a bug-labelled issue** naming the introducing PR via a trailer (for example `Escaped-from: #`); +the issue form is required so a defect that is reported but not yet fixed still counts, rather than staying +invisible until a fix lands. This count MUST be presented as the headline signal and framed as the **review +agent's false-negative rate** (green-then-buggy means the review missed it), and it MUST state that it is only +as complete as the human attribution of the introducing PR. It MUST feed the Correctness dimension. + +#### Scenario: A green-then-buggy PR is counted as an escaped defect +- **WHEN** a merged PR that passed review and tests is later named by a bug report (PR or issue) through the + declared link convention +- **THEN** that PR is counted as an escaped defect, lowers Correctness, and raises the reported review + false-negative rate + +#### Scenario: No escaped defects is distinct from no signal +- **WHEN** enough green merges exist to judge and none is named by a bug report +- **THEN** the escaped-defect count is `0` and Correctness is at its maximum, distinct from the low-signal + "not enough signal" state + +### Requirement: Temporal trend +Beyond the current-vs-to-be radar, the dashboard MUST show how the headline signals move over time: a +per-window series (a small trend line or sparkline) for at least the escaped-defect rate, throughput, and +spec-fit, so an owner can see whether a signal is rising or falling, not only its latest value. Each KPI's +trend indicator MUST be backed by this series. + +#### Scenario: Headline signals carry a trend over windows +- **WHEN** the dashboard renders +- **THEN** the escaped-defect rate, throughput, and spec-fit each show a per-window series, and a KPI's + up/down trend indicator reflects that series + +### Requirement: Minimum signal and progressive reveal +Every metric MUST declare a minimum amount of signal (for example a minimum number of qualifying merged PRs) +below which it does not show a number and instead shows a short note that it is still filling in. A new +project starts mostly in this state; an existing repository that adopts ASDD draws on its historical PR data +and will already clear the bar for most metrics. As signal accrues past a metric's threshold, that metric +starts rendering on its own, with nothing hidden and no misleading value shown in the meantime. + +#### Scenario: A below-threshold metric shows a note, not a number +- **WHEN** a metric has fewer than its minimum qualifying PRs +- **THEN** it shows a short "still filling in" note instead of a value, and begins rendering once the history + reaches the threshold + +#### Scenario: An existing repo clears the bar from history +- **WHEN** a repository with substantial merged-PR history adopts the dashboard +- **THEN** the metrics that its history satisfies render immediately, without waiting for new PRs + +### Requirement: Provenance and privacy unchanged +The analytics section MUST render only facts already visible on GitHub for the repository, MUST keep a private +deployment behind authentication, and MUST require the existing `--public` opt-in before any of it is rendered +for publication. It MUST NOT introduce any new external data source or any new disclosure beyond what the +dashboard already publishes. + +#### Scenario: Public render adds no new disclosure +- **WHEN** the dashboard is rendered with `--public` +- **THEN** every analytics fact shown is one already readable on GitHub by anyone, and no new field is + disclosed that the dashboard did not already publish diff --git a/openspec/changes/project-development-analytics/tasks.md b/openspec/changes/project-development-analytics/tasks.md new file mode 100644 index 0000000..56c58bf --- /dev/null +++ b/openspec/changes/project-development-analytics/tasks.md @@ -0,0 +1,55 @@ +## 1. Producer changes (the two signals the pipeline does not emit today) +- [ ] 1.1 `.github/asdd/intake-check.sh`: expose the spec result (`spec_ok`) as its own field, distinct from + the rolled-up intake pass/fail, so the spec-conformance metric reads the spec signal alone and can + exclude spec-exempt (chore) lanes. +- [ ] 1.2 The review path: emit a machine-readable per-lens result (a per-lens status context, or a parseable + block carrying lens name + verdict + findings) in addition to the human comment, so categories, + malicious, and impact-core are queryable without scraping the comment. Malicious derives from the + security lens verdict (blocking or injected-instruction), not from a label the pipeline never applies. + +## 2. Config and conventions +- [ ] 2.1 Add optional `project.goal` (string) and `project.invariants` (list) to `.asdd.yml`, read by the + dashboard with a no-YAML-dependency scan like the rest of the kit. +- [ ] 2.2 Escaped-defect link convention: `Escaped-from: #` on a `bug`-lane PR OR a bug-labelled + issue, naming the merged PR that introduced the defect. The issue form counts an unfixed escape. + Document it in the PR template and the gates docs. +- [ ] 2.3 Define the core-change marker: a PR touches a declared `protected_path` or the impact lens marks it + normative-core. Note that the MUST-invariant conform/violate metric stays "not enough signal" until a + review emits an explicit invariant check (blocked on a producer, not implied as derivable). + +## 3. Fetch and window +- [ ] 3.1 Extend `fetch()` to read check statuses and changed files for MERGED PRs, not only open ones + (verify merged head-SHA statuses persist after branch delete). +- [ ] 3.2 Compute every metric over an explicit declared window and paginate to cover it, so a repo larger + than one page is never scored over an unstated recent slice. State the window on the page. + +## 4. Analytics model +- [ ] 4.1 `analytics(snap)` in `cli/dashboard.py`: partition PRs into submitted (open = to-be) and merged + (current-state), reusing `stage_of`, `lane_of`, `parse_impact`, `_normative_paths`. +- [ ] 4.2 Topic extraction: rank leading topics per state; map each to the declared goal and invariants with + a plain-language support-or-diverge read. +- [ ] 4.3 Score the six dimensions in `[0, 5]` for current-state and to-be by one rule each. To-be excludes + or down-weights changes-requested PRs. Verification counts only code-changing PRs, Documentation only + user-facing PRs (non-applicable PRs leave the denominator, not scored zero). +- [ ] 4.4 Minimum-signal gate per metric: below its threshold a metric returns "not enough signal", and that + state WINS over a max score (Correctness must not read 5 on an unproven project). +- [ ] 4.5 KPI ratios, escaped-defect rate (as the review false-negative rate), and throughput, each as + total + percentage + failing count, plus a per-window series for the temporal trend. + +## 5. Render +- [ ] 5.1 Two-state summary with per-PR digests. +- [ ] 5.2 Radar/balance map as inline SVG: six labelled axes, 0 centre to 5 point, two overlaid polygons, + a legend, and a "not enough signal" marker on any low-signal axis. No external chart dependency. +- [ ] 5.3 KPI block + the escaped-defect hero (framed as the agent false-negative rate), each with total, + percentage, failing count, and a trend indicator backed by the per-window series. +- [ ] 5.4 Sparklines for the headline signals (escaped-defect rate, throughput, spec-fit); the "still filling + in" note for below-threshold metrics, following the worked HTML design example. + +## 6. Tests and docs +- [ ] 6.1 Extend `cli/dashboard.test.sh`: both states populated and to-be empty; each dimension scored and + the below-threshold path (Correctness not defaulting to 5); escaped-defect linked (PR and issue) and + absent; KPI ratios with and without signal; the window/pagination denominator. Assert the SVG has six + axes and two polygons. +- [ ] 6.2 Update the dashboard docs and setup guide with the new section, the config keys, and the + escaped-defect trailer. Wire the new test into `validation/run-base.py`. +- [ ] 6.3 Confirm `--public` renders only already-public facts and adds no new disclosure.