feat(attention): collect the ranking signals behind the latest-releases leaderboard

[lazizbekravshanov]

Part of #530. Closes #589.

One commit on top of upstream main at 5d9e378.

What this gives the leaderboard

• The "Latest releases" windows fill with attributable numbers. After a clean build of today's main, the 30-day window holds one entry and the 7-day window none, every signal unknown. That is not a data problem: the ranking engine, the benchmark_attention validator and their tests all landed for [12 points] Show the latest leaderboard by default (past 30 days) #530, but nothing produced the block they read. This PR is that producer.
• Every number is dated and points at its exact resource. GitHub stars from the benchmark's own repository, Hugging Face paper upvotes from its paper page, Hugging Face 30-day dataset downloads from its dataset page, each observation carrying observed_at and the resource URL the validator requires.
• No new secret, no workflow change. Both APIs are anonymous for reads; the workflow's existing GITHUB_TOKEN only raises the GitHub budget from 60 to 5,000 requests an hour.
• Cost per daily run, from the committed snapshots as of 2026-09-10: the 30-day window has 312 releases with a dedicated repository, 30 with a paper page and 508 with a dataset page; the 90-day tail adds another 160 repositories and 291 datasets, so a full pass is about 470 GitHub and 830 Hugging Face requests. Budgets default to 800 and 1,200, spent newest release first, and a run stops asking a source after five failures in a row, so an outage costs minutes rather than the timeout times the cohort.

The page half

/leaderboard/ still opens on the model-card adoption view on main. The engine's payload is populated by this PR, and the view that reads it, opening the page on Latest releases · 30d, is the latest-releases view PR. The two are independent and merge in either order: with this one first the page gains real numbers the moment the view lands; with the view first it shows its empty state until this collector runs.

How the collector decides, and what each rule prevents

• Only exact resources. A repository that hosts a benchmark in a subdirectory would credit the host's stars to the benchmark; is_exact_attention_source_url is the single rule, shared with the validator.
• A gone resource is unavailable, never zero. The ranking treats null as "no signal"; a zero would rank a deleted repository below a live one with a single star. HTTP 404, 410 and 451 are the only codes read as facts about the artifact.
• A failed request carries the last reading forward as stale. A rate limit or outage is a fact about the request, not the benchmark. _resolve_attention_metric reports whatever the newest observation says, so writing nothing would leave yesterday's reading reported as fresh; instead the previous snapshot's reading is written again with status: stale and last_successful_date set to the day it was actually read, and a chain of failures keeps that date rather than refreshing it. With nothing to carry forward, nothing is written: a null reading would admit the artifact to the cohort with no signal to rank on. No exception escapes a single resource read (an HTML error page where JSON was expected counts as a failed request), so one bad response cannot abort the hundreds after it.
• Health per (source, metric), ok while anything succeeded. The engine reads a failed health event at the same instant as fresh observations as a reason to demote all of them to stale, so one throttled repository must not flip the whole source; and a paper-page outage must not stale dataset downloads.
• Budgets newest-first, and a pause after consecutive failures. The 7- and 30-day windows the leaderboard shows by default are refreshed before the 90-day tail; a release the budget does not reach keeps its last reading as stale rather than dropping out. A source that fails max_consecutive_failures (5) times in a row is not asked again that run, because a rate limit answers every request alike and waiting out the timeout for each of a thousand resources would outlive the daily workflow. The health row says so: github paused after 5 consecutive failures; 312 resources not read, 298 carried forward as stale.
• Identity follows the engine's rule. Occurrences are grouped by alias-mapped exact artifact key and dated by their earliest released observation, so a re-announcement cannot move a benchmark back into a window it has left, and the observation's canonical_artifact_id is the id filter_release_cohort groups on. Resources are gathered from every occurrence, because the repository of a paper released without one often arrives later on the repository's own updated record; only the date is restricted to released records. A paper and the repository it names are one artifact and cost one request.

Wiring

• src/benchmark_radar/benchmark_attention.py: select_targets, collect_benchmark_attention, merge_benchmark_attention.
• pipeline.run_pipeline: new keyword snapshots (the committed history), because the counters of a three-week-old release keep moving and the collector must see releases older than today's 48-hour lookback; the previous snapshot's block goes in as previous_block for the stale carry-forward; the collector's health rows join the attention health list.
• models.RadarRun.benchmark_attention; snapshots.snapshot_for_run writes the block only when the collector ran, so an absent block means "not collected" rather than "nothing had attention"; merge_snapshots unions same-day passes under the issue Second daily run overwrites the first, dropping items from the permanent corpus #104 rule (a pass that ran out of budget must not erase the first pass's readings, and a reading the later pass only carried forward as stale never replaces one the earlier pass actually made that day).
• cli.py: passes the history in and mirrors the block into out/items.json; briefing.daily_report_run carries the merged day's block.
• config.yml: benchmark_attention block (window_days, per-source max_requests and request_delay_seconds, max_consecutive_failures, attempts, timeout_seconds), with the rules above documented in place.

Tests

tests/test_benchmark_attention.py, fourteen tests, each named for the failure it protects against: identity grouping and exact-resource selection (subdirectory URLs, out-of-window releases and never-released artifacts excluded, a repository learned from an updated record dated by the paper's release); a block the snapshot validator accepts with all three signals as whole numbers; a gone resource as unavailable; a transient failure carrying yesterday's reading forward as stale with the original date, across a chain of days, and writing nothing when there is nothing to carry; the leaderboard reporting that carried reading as stale end to end; a source paused after consecutive failures while the other source is unaffected, and a streak reset by a success; every request failing marking the signal unhealthy; a malformed payload and a non-JSON response as failures that do not abort later reads; budget spent newest-first with the unread tail carried forward; the disabled collector; same-day merge union with the stale-precedence rule; the pipeline passing the history and the previous block through and reporting a merged day's block; and end-to-end, build_latest_releases_leaderboard ranking a release from the collected block that is invisible without it (the #589 failure).

Eleven behaviours were checked by reverting each one and watching its test go red: the carry-forward, the broad exception catch, the pause and its streak reset, resources from updated records, the release date from released records only, the merge precedence and health rules, the carried date, the pipeline passing the previous block, and the integer coercion.

Verification

Clean-worktree run of the CI sequence (git worktree add --detach, submodule initialized): ruff check ., ruff format --check ., normalize-catalog, classify and build-data-release pass; pytest -q reports 1343 passed, 1 failed.

The one failure is tests/test_briefing.py::test_zh_output_budget_covers_worst_case, which downloads tiktoken's o200k_base table from openaipublic.blob.core.windows.net, a host the build sandbox's egress policy blocks. It fails identically on an untouched main checkout in the same sandbox and passes on GitHub's runner.

The live GitHub and Hugging Face endpoints were not reachable from the sandbox, so the response fields (stargazers_count, upvotes, downloads) follow the documented payloads and the connectors in sources.py that already read them. The first daily run after merge is the first live exercise; the three new rows in source health will show any surprise.

Not in this PR

The /leaderboard/ default view (the latest-releases view PR, see above). Paper upvotes are read only where an artifact already carries a Hugging Face paper URL (30 of the 1,283 targets in the 90-day cohort); deriving a paper page from every arXiv id would cost a request per paper for mostly 404s, and is a separate decision. Fallback counters from connector metrics keep their existing unknown status.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UaUf4LgR9HogJ1DhXwjESy

[Copilot]

🟡 Changes recommended

Unresolved moderate collector issues affect API correctness, budgets, authentication, and ranking behavior.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a daily collector for GitHub and Hugging Face attention signals powering the Latest Releases leaderboard.

Changes:

• Collects, validates, budgets, and persists dated attention metrics.
• Wires attention data through pipelines, snapshots, CLI output, and briefings.
• Adds configuration and comprehensive tests.

File summaries

File	Summary	Final review notes
tests/test_benchmark_attention.py	Tests collection, persistence, failure handling, merging, and ranking.	Nit (1 vote): preserve ranking eligibility gates for unreviewed releases.
src/benchmark_radar/snapshots.py	Persists and merges attention blocks.	No final findings.
src/benchmark_radar/pipeline.py	Supplies history and collects signals.	No final findings.
src/benchmark_radar/models.py	Adds attention data to RadarRun.	No final findings.
src/benchmark_radar/cli.py	Passes history and exports attention data.	No final findings.
src/benchmark_radar/briefing.py	Carries merged attention data into reports.	No final findings.
src/benchmark_radar/benchmark_attention.py	Implements metric collection and merging.	Moderate findings: normalize .git repository URLs, reject fractional counters, emit unknown observations after first-read failures, count retry attempts against budgets, and send the actual GitHub token.
config.yml	Configures collection budgets and retries.	No final findings.

Review details

Suppressed comments (5)

src/benchmark_radar/benchmark_attention.py:209

• is_exact_attention_source_url explicitly accepts repository clone URLs ending in .git (the existing leaderboard test covers this), but the API path below uses that suffix verbatim. A resource such as https://github.com/org/bench.git will therefore request /repos/org/bench.git and be recorded as unavailable instead of collecting the repository's stars. Strip the clone suffix when building the API path while retaining the original URL for attribution.

        owner, repo = segments[0], segments[1]
        payload = get_json(
            f"https://api.github.com/repos/{owner}/{repo}",

src/benchmark_radar/benchmark_attention.py:375

• When a first-time resource read fails, this branch omits the observation entirely. The release leaderboard uses the presence of a dated attention observation to keep a candidate in the cohort, so a transient outage makes this release disappear instead of leaving it visible with an unknown signal/limited status. Emit a null unknown observation here so missing measurements do not remove the record.

            carried = previous.get((signal, url))
            if carried is None:
                continue

src/benchmark_radar/benchmark_attention.py:354

• requests_made[source] is incremented once per resource, but _read_resource passes the configured attempts through to get_json; with the production value of 2, one failed resource can consume two real API requests. The 800/1,200 max_requests settings and the cost/rate-limit claims therefore do not cap actual requests, especially during an outage. Count retry attempts against the source budget or make the budget's resource-read semantics explicit and enforce a separate HTTP-request cap.

                if requests_made[source] and delays[source]:
                    sleep(delays[source])
                requests_made[source] += 1
                reading = _read_resource(
                    signal, url, options=options, get_json=get_json, stats=signal_stats

src/benchmark_radar/benchmark_attention.py:211

• _github_headers() currently emits the literal Authorization: ****** whenever GITHUB_TOKEN is set, so this new collector will still call GitHub anonymously and hit the 60-request limit instead of the documented authenticated budget. Update the shared helper to send the actual bearer token (without exposing it in errors) before relying on it here.

        payload = get_json(
            f"https://api.github.com/repos/{owner}/{repo}",
            headers=_github_headers(),
            **options,

tests/test_benchmark_attention.py:657

• With reviewed_benchmark_ids=set(), this test makes a fresh collector observation sufficient for a formal rank. filter_release_cohort treats any benchmark_attention observation as has_dated_attention, so keyword-selected, non-reviewed releases bypass #530's rule that discovery recall alone must remain unranked until verification. Keep this case limited/unranked or add the review/verification gate before attention observations establish ranking eligibility.

    payload = build_latest_releases_leaderboard(snapshots, as_of=NOW, reviewed_benchmark_ids=set())

    window = payload["windows"]["30d"]
    assert window["ranked_count"] == 1
    entry = window["entries"][0]
    assert entry["status"] == "ranked"
    stars = entry["components"]["github_stars"]
    assert (stars["value"], stars["status"], stars["source_url"]) == (150, "fresh", REPO)

• Files reviewed: 8/8 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[ktwu01]

Thanks for the careful collector work. I found two correctness issues that need a small follow-up before merge:

• The failure breaker is shared by provider, so five Hugging Face paper failures pause dataset-download collection too. I reproduced the dataset request being skipped, its old value becoming stale, and its health row still saying ok: true. Please key pause and streak state by signal while keeping the shared request budget, and cover the cross-signal case.
• _counter still accepts fractional stars, upvotes, and downloads as fresh values. These APIs expose whole-number counters, so non-integral payloads should fail validation.

The full clean CI sequence passes both on this head (1,378 tests) and merged with current main (1,400 tests); there are no merge conflicts. Thank you.