Skip to content

feat(countme): implement weekly countme client reporting to countme.projectbluefin.io - #145

Closed
mrbobbytables wants to merge 3 commits into
projectbluefin:mainfrom
mrbobbytables:feat/countme-reporting
Closed

mrbobbytables wants to merge 3 commits into
projectbluefin:mainfrom
mrbobbytables:feat/countme-reporting

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Summary

Implements weekly countme client reporting to countme.projectbluefin.io following Project Bluefin ADR 0006 and addresses all review feedback from #69.

Changes

  1. Client Script: Added /usr/libexec/utah-countme to:
    • Extract image metadata (repo, tag, flavor, arch) from /usr/share/ublue-os/image-info.json or fallback to /etc/os-release.
    • Maintain install epoch in /var/lib/utah-countme/epoch and calculate Fedora-style cohort age buckets (1: first week, 2: 2–4w, 3: 5–24w, 4: >24w).
    • Degrade cleanly without resetting epoch if the epoch file is corrupted or unreadable.
    • Throttle pings to once every 7 days via /var/lib/utah-countme/lastrun. If curl fails, lastrun is left unwritten to allow retry on the next schedule without dropping a weekly cycle.
    • Send empty GET ping to https://countme.projectbluefin.io/metalink?repo=${REPO}&tag=${IMAGE_TAG}&flavor=${IMAGE_FLAVOR}&arch=${ARCH}&countme=${BUCKET} with a --max-time 10 timeout.
    • Honor opt-out via /etc/projectbluefin/countme/disabled or timer masking.
  2. Systemd Units:
    • Added utah-countme.service (DynamicUser=yes, StateDirectory=utah-countme, ExecStart=/usr/libexec/utah-countme).
    • Added utah-countme.timer (OnBootSec=10m, OnCalendar=weekly, Persistent=true, RandomizedDelaySec=12h).
  3. Preset & Contract:
    • Enabled utah-countme.timer in system_files/shared/usr/lib/systemd/system-preset/85-utah-desktop.preset and scripts/configure-services.sh.
    • Added utah-countme.timer to contracts/bluefin-desktop.toml under [services].enabled.
    • Added utah-countme assertions to Justfile's check recipe.
  4. Test Suite:
  5. Documentation & Skill Catalog:
    • Documented transmitted query string parameters, empty GET payload, aggregate server logging, network resilience, and opt-out paths in docs/skills/desktop-contract.md.
    • Updated skill version to 1.1 and regenerated docs/skills/index.json and docs/skills/index.md.

Verification

  • Endpoint readiness verified: curl -s "https://countme.projectbluefin.io/metalink?repo=utah&tag=latest&flavor=main&arch=x86_64&countme=1"' returns countme accepted for repo=utah tag=latest flavor=main gamemode=0 arch=x86_64 countme=1`.
  • All 89 unit tests pass via just test.
  • Full static gate passed via just check.

Closes #49

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 196f378

…rojectbluefin.io

Implement weekly client reporting to countme.projectbluefin.io following
Project Bluefin ADR 0006.

- Add /usr/libexec/utah-countme script to compute weekly cohort bucket,
  extract image metadata (repo, tag, flavor, arch), and send empty GET ping.
- Add systemd units utah-countme.service and utah-countme.timer with
  DynamicUser=yes and StateDirectory=utah-countme.
- Enable utah-countme.timer in 85-utah-desktop.preset and configure-services.sh.
- Add utah-countme.timer to contracts/bluefin-desktop.toml.
- Support opt-out via /etc/projectbluefin/countme/disabled and timer masking.
- Add comprehensive unittest suite in tests/test_countme.py.
- Document telemetry query params, logging policy, resilience, and opt-out
  in docs/skills/desktop-contract.md and regenerate skill catalog.

Closes projectbluefin#49

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 196f378)

  1. Medium — weekly throttle interacts badly with timer jitter, dropping ~1 in 3 weekly pings. utah-countme.timer uses OnCalendar=weekly with RandomizedDelaySec=12h and no FixedRandomDelay= (system_files/shared/usr/lib/systemd/system/utah-countme.timer:9-13), so each week's fire time is Mon 00:00 + dₙ with dₙ re-rolled per activation. The script skips when NOW - LASTRUN < 604800 strictly (system_files/shared/usr/libexec/utah-countme:52-56, WEEK=604800 at :11). Whenever the new delay is smaller than last week's (~50% of activations), the gap is under 7 days and the ping is silently skipped, then sent the following week — a steady-state undercount of roughly a third of weekly reports. Fix by adding FixedRandomDelay=yes to the timer, or throttling with slack (e.g. NOW - LASTRUN < WEEK - 12h), or comparing week indices instead of a strict interval. The PR body's claim of "once every 7 days" throttling without "dropping a weekly cycle" does not hold as written.

  2. Low — query parameters are not URL-encoded. REPO, IMAGE_TAG, and IMAGE_FLAVOR are interpolated raw into the URL (system_files/shared/usr/libexec/utah-countme:130). Values come from image-info.json / os-release, so a tag containing +, space, or & (plausible in custom variants) would corrupt or split the query string. curl --data-urlencode -G or a small encode helper would close this.

  3. Info — test gap on the throttle boundary. tests/test_countme.py:169-194 covers "1 day ago → skip" but not the boundary/jitter case from finding 1; a test with lastrun = now - (WEEK - hours) would have caught it.

Bucket math (utah-countme:59-77), corrupt-epoch degradation, curl-failure lastrun handling, and the os-release fallback all check out and are well covered by tests.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@Danathar

Copy link
Copy Markdown
Contributor

Before this goes further: projectbluefin/common#1112 adds the same bluefin-countme service/timer/preset/script to common's system_files/shared, and Utah's Containerfile copies that tree in (Containerfile:66-67). Once #1112 merges and the common pin moves, its files overwrite the ones here, leaving a divergent copy nobody maintains. dakota#1561 is already switching to the shared client. Suggest reducing this PR to the Utah-side wiring only (enable the timer, test asserts, live-ISO disable) and sequencing it after common#1112 — same conclusion I reached on bluefin#1247.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is close, and most of what I went looking for holds up. I checked a few things that looked like bugs and were not, so for the record: [ -e "$DISABLED_FILE" ] && exit 0 is errexit-safe (the test is the left side of the final &&, so it is exempt and the script continues); and the grep ... | cut | tr ... || echo "unknown" fallbacks do fire, because set -o pipefail propagates grep's non-match — I ran both. image-info.json really is written by scripts/configure-branding.sh with exactly the three keys the script reads (image-name, image-tag, image-flavor), so ConditionPathExists on it is satisfiable and the os-release fallbacks are dead paths in practice rather than wrong ones. And the endpoint is live — that URL shape returns HTTP 200 today. The Fedora bucket mapping (1 / 2–4 / 5–24 / >24, 1-based weeks) matches upstream, and the NOW < EPOCH clock-skew guard is a nice touch.

Two things need changing before this goes in.

1. jq is not in the package contract, so the reporter is a silent permanent no-op.

The script hard-requires jq, and on a missing jq it logs one line to the journal and exits 0. Neither jq nor curl appears in any section of packages/bluefin.toml or packages/utah.toml:

$ python3 -c "...scan every [section].packages in both manifests for jq/curl..."
(no output)

So whether this feature works at all depends entirely on whether the Hummingbird base happens to ship jq, which nothing in this repo asserts and nothing here would notice changing. The failure mode is the worst kind for telemetry: it looks healthy, the timer runs weekly, and the count is zero forever. system_files/shared/usr/libexec/bluefin-refresh-stats has the same latent dependency, which is an argument for fixing it properly rather than inheriting the gamble.

Either add jq to the contract with a comment saying the countme client needs it — and a test asserting it, since the contract is the thing that is machine-checked — or drop the dependency. Dropping is easy here: the script reads three string fields out of one JSON file, and python3 is unambiguously present (every build script in this repo is python3). A python3 -c one-liner, or just reading the values without a JSON parser, removes the question entirely. I mildly prefer that, because it also removes the command -v early-exit and lets a genuinely broken image-info.json be loud.

2. The throttle only advances on a successful request, which turns any endpoint outage into a per-boot retry storm.

if curl -sf --retry 3 --max-time 10 -H "User-Agent: ${USER_AGENT}" "$URL" >/dev/null; then
    echo "$NOW" > "$LASTRUN_FILE"
fi

With OnBootSec=10m on the timer, a machine that is rebooted daily will re-attempt every single boot for as long as the endpoint is down or the network is captive, each time with --retry 3. The endpoint is up right now, but a weekly counter is precisely the kind of service that will have an outage window, and this is the behaviour that makes an outage look like a traffic spike when it recovers. Fedora's own dnf countme does not retry: the bucket rides along with a metadata refresh and a failed one is simply lost, because for a population counter a dropped sample is cheap and a duplicated one is not. Write $LASTRUN_FILE unconditionally, before or regardless of the request, and let a failed week go uncounted.

Two smaller notes, non-blocking:

  • The epoch is "first time this script ran" rather than install time. That is a fine proxy (10 minutes after first boot), but it is worth a comment saying so, because it means the age bucket is measured from first successful timer run — a machine whose first boot had no network still gets a sensible epoch, but one where /var/lib/private was wiped silently resets to bucket 1 and looks like a new install. Not worth engineering around; worth writing down.
  • ConditionPathExists=!.../disabled on the timer is only evaluated when the timer starts, so a user creating the opt-out file will not stop the timer until reboot. Harmless, because the service has the same condition and the script checks it too — three layers deep is fine — but the opt-out path itself is not documented anywhere in the repo. Since this is on-by-default telemetry, the exact touch /etc/projectbluefin/countme/disabled incantation should appear in the README or the docs page that Documentation= points at, not only in a comment in the script.

The branch conflicts with main as it stands, so it needs main merged in before CI can speak to it.


Generated by Claude Code

mrbobbytables and others added 2 commits September 19, 2026 15:51
# Conflicts:
#	Justfile
#	docs/skills/desktop-contract.md
#	docs/skills/index.json
#	docs/skills/index.md
…and document opt-out

Address review feedback from @hanthor:
- In utah-countme, replace jq invocation with inline python3 json parsing,
  removing the uncontracted runtime dependency on jq.
- Advance the throttle by writing lastrun unconditionally before the ping
  to avoid retry storms across reboots during outages.
- Document epoch semantics and add countme opt-out instructions to README.md.
- Update test suite to verify lastrun is written regardless of curl exit status.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback in commits 45bbc0b (merge main) and 3c2dff9:

  • Merged current main and resolved conflicts in Justfile and docs/skills/desktop-contract.md.
  • Replaced the jq dependency in system_files/shared/usr/libexec/utah-countme with standard library python3 -c parsing, removing uncontracted dependency and allowing malformed JSON to fail loud.
  • Advanced throttle by recording lastrun unconditionally before attempting the request, preventing per-boot retry storms during endpoint or network outages.
  • Documented epoch initialization semantics as a proxy for install age cohorting.
  • Added explicit opt-out documentation (touch /etc/projectbluefin/countme/disabled and timer masking) to README.md.

🐝 Hive Agent: contributor | SHA: unknown

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I think is still open from the earlier automated review, separate from the two points hanthor raised.

The weekly throttle and the timer jitter work against each other. utah-countme.timer fires OnCalendar=weekly with RandomizedDelaySec=12h and no FixedRandomDelay=, so each Monday's run lands at 00:00 plus a fresh random delay of up to 12 hours. The script then skips when NOW - LASTRUN is strictly less than 604800 (the "Throttling" block in system_files/shared/usr/libexec/utah-countme). Whenever this week's delay is shorter than last week's, which is about half the time, the gap is under seven days and the ping is skipped. It goes out the following week instead. On a machine that stays up, that is roughly one weekly report in three lost. And now that lastrun is written before the request, a later OnBootSec run in the same week cannot recover it.

I reproduced it at 3c2dff9 with lastrun set to now minus 7 days plus 3 hours: the script exits 0, curl is never called, and lastrun is unchanged.

Any of these would close it:

  • compare week numbers, floor(NOW / 604800) against floor(LASTRUN / 604800), which is what dnf's countme does;
  • throttle with slack, for example NOW - LASTRUN < 604800 - 13*3600;
  • add FixedRandomDelay=yes to the timer (on its own AccuracySec=1h can still shave the gap, so I would pair it with the slack).

A test with lastrun = now - (WEEK - a few hours) that expects a ping would pin it.

Minor and non-blocking: repo, tag and flavor are interpolated into the URL without encoding. Today's values from configure-branding.sh are safe, but a tag containing + or & would split the query string.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — default-on telemetry encoded as a build-time contract invariant needs maintainer sign-off (@mrbobbytables); #69 was closed asking exactly for that confirmation and an ADR 0006 link, and this re-roll still enables by default without linking the ADR.

intent-alignment

  • This PR re-lands the default-on posture that @hanthor's #69 review explicitly said "needs a maintainer sign-off, not a code review": the timer is preset-enabled (system_files/shared/usr/lib/systemd/system-preset/85-utah-desktop.preset:9), enabled in scripts/configure-services.sh:72, and made a contract invariant in contracts/bluefin-desktop.toml:141 — an image without telemetry now fails verification. ADR 0006 is referenced (PR body, docs/skills/desktop-contract.md:117) but still not linked or quoted, which #69 asked for.
  • PR body contradicts the code: it claims "If curl fails, lastrun is left unwritten to allow retry on the next schedule." The script writes lastrun unconditionally before the request (system_files/shared/usr/libexec/utah-countme:134-136), and the test asserts exactly that (tests/test_countme.py, test_curl_failure_still_updates_lastrun_to_prevent_retry_storm). The code/docs behavior is a defensible choice, but the body describes the opposite one — please fix the description so the record matches what ships.

correctness

  • Cadence undercount: the 7-day throttle (utah-countme:47-52) combined with OnCalendar=weekly (utah-countme.timer:9-13) means any ping fired mid-week (e.g. via OnBootSec=10m + RandomizedDelaySec=12h) causes the next Monday firing to be skipped, so always-on systems can settle into ~fortnightly reporting. Fedora's countme avoids this with a window check; using a threshold slightly under a week (e.g. 6 days) would too. Undercounting is directly counter to the feature's purpose.

docs-currency

  • The documented opt-out command fails: README.md:106-108 says sudo touch /etc/projectbluefin/countme/disabled, but nothing in the image ships /etc/projectbluefin/countme/ (no tmpfiles.d entry; only utah-tuned.conf exists) — touch will error with ENOENT. Docs should say sudo mkdir -p /etc/projectbluefin/countme && sudo touch .../disabled, or the image should ship the directory. Same command implied in docs/skills/desktop-contract.md:135.

style

  • The /etc/os-release fallback (utah-countme:104-131) is dead in production: both units gate on ConditionPathExists=/usr/share/ublue-os/image-info.json (utah-countme.service:7, utah-countme.timer:5), so the fallback only runs in tests. Fine as defense-in-depth, but the PR body/docs present it as a live path.

No findings from: security.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff: implements weekly countme client reporting. Blocked on timer jitter and review findings in utah-countme.timer. Needs author revision.

@castrojo

Copy link
Copy Markdown
Contributor

Closing as a product/telemetry policy decision: default-on weekly countme telemetry requires maintainer architectural sign-off and an ADR rather than an automated code change, in addition to timer jitter redesign.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(countme): implement weekly countme client reporting to countme.projectbluefin.io

4 participants