Skip to content

CIMetrics: report ccache hit rates with --ccache - #48

Open
willson556 wants to merge 3 commits into
mainfrom
ci-metrics-ccache
Open

willson556 wants to merge 3 commits into
mainfrom
ci-metrics-ccache

Conversation

@willson556

Copy link
Copy Markdown
Member

The shared ccache store rollout (CI RFC Proposal 2) gates each step on ccache hit rates and on how many jobs start with an empty cache. The Sep 17–23 baseline for those came from a one-off log scrape (ccache_stafllib.md). This adds them to CIMetrics so every gate uses the same script.

--ccache:

  • Downloads the log of every completed job that has a step with "ccache" in its name, and parses the last ccache -s in it.
  • Handles both layouts in use: Ubuntu's ccache 4.5 (Summary:) and the IAR-capable fork, 4.12 (Cacheable calls:). It reads the first Hits line, which counts each cacheable call once, not the per-storage lookup counts below it.
  • Also reads remote-storage hits, misses, errors and timeouts, for once jobs use the shared store.
  • Caches only the parsed stats, under <out>/<repo>/ccache/. Logs that can't be fetched are retried on the next run.

Report: a ccache section per repo, with jobs measured, jobs with no hits, share with no hits, and hit rate (hits over cacheable calls, summed):

  • by job, labelled by build preset (build-embedded (arm-cm7-iar-Debug)). GitHub cuts job names at 100 characters, so the label doesn't depend on the closing parenthesis.
  • by PR state at run time and the PR's base: on main/dev, or stacked. The base is the PR's base now, so older weeks undercount stacked runs; the README says so.
  • remote-storage errors and timeouts, once there are any.

jobs.tsv gains job_id, base, ccache_hits and ccache_cacheable columns.

Fix along the way: the PR cache was <out>/<repo>/prs.json regardless of date range, so a second range reused the first range's PR list. It's now prs_<since>_<until>.json.

Testing:

  • uv run --with pytest pytest CIMetrics: 9 cases. They cover both stats layouts (copied from real Sep 18 logs), the remote-storage block, ANSI codes, a log with no stats, job labels including a cut-off name, and the row math. A new ci-metrics-tests.yml workflow runs them.
  • Ran --ccache on Sep 23 for StaflLib and coit-tower. All 361 logs were fetched and parsed, and none failed.
  • All 12 jobs of StaflLib run 35803238245 match the old scrape exactly (e.g. host-Debug 446 / 909, every embedded build 0 hits).
  • The Sep 23 numbers agree with the RFC: StaflLib embedded builds started with an empty cache in 44–50% of jobs, while host test jobs never did.

Follow-up: once SetupCcache exists, it could print its stats as a ::notice::. CIMetrics could then read them from the check-run annotations instead of downloading roughly 1,800 logs a week.

🤖 Generated with Claude Code

The shared ccache store rollout gates on per-job hit rates and on how
many jobs start with an empty cache. The Sep 17-23 baseline for those
came from a one-off log scrape; this puts it in CIMetrics so every gate
uses the same script.

--ccache downloads the log of each completed job with a ccache step,
parses the last `ccache -s` (ccache 4.5 and 4.12 layouts, plus remote
storage errors and timeouts), and adds a ccache section per repo: by job,
and by PR state and base (main/dev or stacked). jobs.tsv gains
ccache_hits and ccache_cacheable columns. Only parsed stats are cached.

Also fixes the PR cache, which was keyed by repo only, so a second date
range reused the first range's PR list. It is now keyed by range.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@willson556
willson556 marked this pull request as ready for review October 9, 2026 17:45
@staflsystemsci
staflsystemsci Bot requested review from a team, fcheein, gretl-silva-stafl, mament0, mns4595, mzhu48, nehalkpatel and thebhef and removed request for a team October 9, 2026 17:45
willson556 and others added 2 commits October 9, 2026 10:55
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@staflsystemsci

staflsystemsci Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Review SLA

Nobody owes a review on this PR right now.

Due times are in business hours, 10:00 to 17:00 Pacific: 4 h for a PR under 250 added lines, 7 h otherwise, and 4 h for a re-review. Mid-stack, an assignee owes a response on the lowest PR they haven't approved. Dashboard

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants