Skip to content

feat: add truthful benchmark phase telemetry - #304

Merged
shaggitza merged 4 commits into
mainfrom
fix/283-benchmark-incremental
Jul 29, 2026
Merged

feat: add truthful benchmark phase telemetry#304
shaggitza merged 4 commits into
mainfrom
fix/283-benchmark-incremental

Conversation

@shaggitza

Copy link
Copy Markdown
Owner

Summary

  • publish schema-v4 runner manifests with exact preparation and cold-build phase boundaries
  • mark warm, incremental, RSS, and cache-size telemetry explicitly not measured
  • authenticate producer/evaluator status, timing keys, phase order, aggregates, and unresolved normalization fail closed
  • preserve schema-v3 compatibility and prevent writes into frozen benchmark artifacts/results

Validation

  • 62 focused runner/evaluator tests passed
  • adversarial complete-manifest, cleanup-failure, phase-order, extra-timing, and frozen-path cases passed
  • Ruff, strict mypy, clean-index and frozen-artifact checks passed
  • four independent review rounds; final verdict PASS

Partial progress on #283; real warm/incremental backend reuse, RSS/cache measurement, and remaining release-attestation gates stay open.

@shaggitza
shaggitza merged commit f812c36 into main Jul 29, 2026
6 checks passed
@shaggitza
shaggitza deleted the fix/283-benchmark-incremental branch July 29, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants