Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions docs/development/community-metrics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Community metrics

Issue [#2554](https://github.com/VMAFx/vmafx/issues/2554) (maintainer decision
Q-182 item d): response times and contribution flow are measured, not guessed.
`scripts/dev/community_metrics.py` produces the numbers; **output only** —
nothing is posted anywhere, the maintainer decides what to publish.

## Metrics

For a date window `--since` .. `--until` (inclusive):

| Metric | Definition |
| --- | --- |
| Opened / closed | Issues and pull requests created or closed inside the window (a pair both opened and closed inside the window is counted once, in both groups). |
| Maintainer first response | For every item **opened** in the window: the earliest comment or review authored by a maintainer — the `@logins` of [MAINTAINERS.md](https://github.com/VMAFx/vmafx/blob/master/MAINTAINERS.md) — that is not authored by a bot. Reported as the median and the 90th percentile in days, over the items that have one. An item with none is a stated `no response yet` row — never zero, never dropped. A response arriving after the item was closed still counts and its row is marked `post-close`. |
| Bots | Every login ending in `[bot]` seen in the window or the merged history. They are excluded from first responses and first merges, and the report lists the logins it treated as bots so the rule is auditable. |
| Open `good first issue` / `help wanted` | Current open counts for the two labels (a snapshot, not window-scoped). |
| Aging pull requests | Open pull requests older than 30 days, with their age. |
| First merged pull request | Authors whose **earliest merge** falls inside the window, derived from the whole merged history — never from `author_association`, which GitHub computes at query time (a merged pull request's author always reads `CONTRIBUTOR`; measured on #1082, 2026-10-21). Bots and maintainers are excluded. The merged-history fetch fails when it reaches its page limit instead of truncating, because a truncated history would misreport exactly this number. |

## Running it

```bash
python3 scripts/dev/community_metrics.py --since 2026-10-01
```

`--until 2026-10-31` bounds the window's end (default: today) and
`--format json` switches the report to JSON. `--repo OWNER/REPO` overrides
the repository (default: the `gh` current repository). The script reads the
GitHub API through the `gh` CLI — no token handling of its own — and prints
the report to stdout in Markdown (default) or JSON.

## Tests

The fixture suite runs the whole pipeline on recorded API response shapes with
no network:

```bash
python3 -m unittest scripts.dev.tests.test_community_metrics
```

The fixtures cover the acceptance cases of issue #2554: expected medians and
counts, an item with no maintainer response (stated, never zero), an empty
window, a single item, an answer arriving after close, a pull request whose
first response is a plain conversation comment, an aging pull request created
outside the window, the first-merge derivation ignoring `author_association`,
and every listing failing at its result cap instead of truncating.
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -491,6 +491,7 @@ nav:
- Container base images: development/base-images.md
- SYCL zero-copy testing: development/sycl-zerocopy-testing.md
- Preflight (local CI gate): development/preflight.md
- Community metrics: development/community-metrics.md
- Default model: development/default-model.md
- Release: development/release.md
- Inherited Netflix tags: development/inherited-tags.md
Expand Down
Loading