Repository navigation
ci(measurement): object-level evidence model (report-only), mutation-policy/2 - #174
Merged
sinkog merged 2 commits intoNov 14, 2026
Merged
Conversation
…policy/2 Global percentages do not say which production function got which check. This adds an object-level, verifiable measurement model. It is report-only: nothing gates on it, nothing is skipped because of it, no threshold or baseline changes. mutation-policy/2 (owner decisions): the primary Rust score is caught/(caught+missed); an unviable mutant is neither caught nor missed and is a mandatory metric of its own (count and ratio); missed and timeout still fail, unviable still does not. A zero denominator is neither 100% nor 0%: NOT_APPLICABLE (NO_MUTANTS_PLANNED, ALL_UNVIABLE, NO_EVALUABLE_MUTANT) or UNVERIFIABLE (MEASUREMENT_MISSING). The old caught/planned value stays printed as `legacy`. scripts/object_model.py (object-id/1, object-evidence/1): - Go objects from tools/objinventory (new, stdlib-only, separate module like goastdump): exact function spans. A cover block is attributed only if it lies completely inside one object; a block that cuts an object, spans several, or lies in none is counted and the affected fields are UNVERIFIABLE. Nothing is split proportionally. Build-tag variants are separate objects. Repeated profiles mark an object whose coverage differs as NONDETERMINISTIC. - Rust objects from the real cargo-llvm-cov export, named with the binutils c++filt of the pinned image (recorded in the record). Instances (generics, closures, per-binary copies) fold into one source object; the instance detail is kept. Test code is found by tokenizing the #[cfg(test)] item span, not by "everything after the first #[cfg(test)]" (that dropped production code: measured on ffi/src/lib.rs, caught by the L3 check below). - Mutation verdicts are mapped to objects by file and the last segment of the (module-qualified) function name; a mutant that fits two objects is ambiguous and counted, never guessed. - Every evidence field is MEASURED, NOT_APPLICABLE or UNVERIFIABLE with a coded reason. The coverage kind is statement, line or region: there is no "branch" (stable rust has none). - The record is canonical and digested; the offline verifier reports three separate levels: L1 INTEGRITY_OK, L2 BOUND_TO_INPUTS (NOT_CHECKED without inputs), L3 COMPLETE_FOR_SCOPE (PARTIAL where only a part can be shown). There is no combined "verified". .measurement-scope.json (measurement-scope/1): every tracked Go/Rust file is production, test, test-data, experimental, a separate module or a tool, each exclusion with a reason; an unclassified file is an error. tools/goastdump is measured and reported on its own. .github/workflows/object-evidence.yml: report-only; not a required check; nothing needs it. verify.yml: the objinventory module is vetted and tested in the python job. mutation_plan.py: object_model.py and the scope file are gate logic. Tests: 43 on real tool output (N-group: new contracts) plus the policy tests (R-group). The N tests were shown to have teeth by removing each check on a scratch copy. --- [signing-metadata] key = cic-my-sign-key signature = vault:v1:MEYCIQChB8E3dos3hN3/2pwcRYLhIyYTDu8ON37MEiAZZZxZSgIhAO6zWiuAfaKju9UISjk0ub0i3xzhNj1QY1UP+e2jFnba hash-algorithm = sha256 digest = LvHUz6xpK/g717aNSbrrW9tk5HxOXimd8tArCeFD6MI= [certificate] -----BEGIN CERTIFICATE----- MIICBjCCAaygAwIBAgIUSnRMR6RPnEbg296XWPOqq/u5PCwwCgYIKoZIzj0EAwIw QzELMAkGA1UEBhMCSFUxGTAXBgNVBAoMEENlbnRyYWxJbmZyYUNvcmUxGTAXBgNV BAMMEENJQyBEZXZlbG9wZXIgQ0EwHhcNMjYwMzIwMTMyMjU5WhcNMjYxMjMxMTMy MjU5WjBFMQswCQYDVQQGEwJIVTEZMBcGA1UECgwQQ2VudHJhbEluZnJhQ29yZTEb MBkGA1UEAwwSR2Fib3IgWm9sdGFuIFNpbmtvMFkwEwYHKoZIzj0CAQYIKoZIzj0D AQcDQgAEIG2CVmTfmLB9pLLclj7YmP2eedAjklpy4LGrU2ijoiy6Xqpuybv7OgJe i+ez31s65NEV8+X/ByeX1cstR988z6N8MHowCQYDVR0TBAIwADAdBgNVHQ4EFgQU yZN6AIX/TNnIJ9GwAa/NRN3ujHAwHwYDVR0jBBgwFoAUXn6CHYzPUqU4JVP8g+OS WeDYjhcwDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEF BQcDBDAKBggqhkjOPQQDAgNIADBFAiEA+bFzXRoJ4PCQbhAAtpkcMjt0vNj5rEW0 lOMBGDNyaWkCIB1vmM7PcZzv/c9bIrxF5kqv6QXomouhByUfeNUTbpKW -----END CERTIFICATE-----
…e scope The first CI run of this change failed in `coverage` and in `go (chunk 4)`/`completeness`/ `mutation-pr`: the fixture `tests/test_tools/fixtures/object_model/goastdump-main.go` is a `package main` file, which the root module treats as a package of its own (centralrelay/tests/test_tools/fixtures/object_model). The coverage floor (0% < min 80%) and the mutation gate (0 of 37 mutants covered, efficacy 0.00) both failed on it. Not an environment problem and not pre-existing: the base has no such file; it is a regression of this change. The same mistake showed in the measurement scope: `*.go` classified the fixture as production, so the object-evidence run counted 479 root objects instead of 464 (the 15 functions of the fixture). - the Go and Rust fixture sources are stored as .txt (no tool sees source code in them) - .measurement-scope.json: tests/* is test-data for Go and Rust, with a reason - two guard tests: no .go/.rs file under tests/ outside testdata; a source file under tests/ is not production in the scope No blocking gate, threshold or baseline is touched. --- [signing-metadata] key = cic-my-sign-key signature = vault:v1:MEUCIQDz9d60E5aPqHDufCfNCgFmkm1pqVyKaRgR6+MK9wxqeAIgErThcuAT975EUDLfLUfJHtqar5RC0Bb287v7eEH14kI= hash-algorithm = sha256 digest = xuua4oLhoRGpKfSz4kPhgBci7dYy/qVTC1HcoWoeoI8= [certificate] -----BEGIN CERTIFICATE----- MIICBjCCAaygAwIBAgIUSnRMR6RPnEbg296XWPOqq/u5PCwwCgYIKoZIzj0EAwIw QzELMAkGA1UEBhMCSFUxGTAXBgNVBAoMEENlbnRyYWxJbmZyYUNvcmUxGTAXBgNV BAMMEENJQyBEZXZlbG9wZXIgQ0EwHhcNMjYwMzIwMTMyMjU5WhcNMjYxMjMxMTMy MjU5WjBFMQswCQYDVQQGEwJIVTEZMBcGA1UECgwQQ2VudHJhbEluZnJhQ29yZTEb MBkGA1UEAwwSR2Fib3IgWm9sdGFuIFNpbmtvMFkwEwYHKoZIzj0CAQYIKoZIzj0D AQcDQgAEIG2CVmTfmLB9pLLclj7YmP2eedAjklpy4LGrU2ijoiy6Xqpuybv7OgJe i+ez31s65NEV8+X/ByeX1cstR988z6N8MHowCQYDVR0TBAIwADAdBgNVHQ4EFgQU yZN6AIX/TNnIJ9GwAa/NRN3ujHAwHwYDVR0jBBgwFoAUXn6CHYzPUqU4JVP8g+OS WeDYjhcwDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEF BQcDBDAKBggqhkjOPQQDAgNIADBFAiEA+bFzXRoJ4PCQbhAAtpkcMjt0vNj5rEW0 lOMBGDNyaWkCIB1vmM7PcZzv/c9bIrxF5kqv6QXomouhByUfeNUTbpKW -----END CERTIFICATE-----
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-04 of #169. A report-only, object-level measurement model. No new blocking gate, no test skipping, no threshold, floor or baseline change (
.mutation-baseline.jsonandscripts/gremlins_gate.pyare untouched in the diff).What it adds
mutation-policy/2(owner decisions): the primary Rust score iscaught/(caught+missed); anunviablemutant is neither caught nor missed and is a mandatory metric of its own (count and ratio), it does not fail;missedandtimeoutstill fail. A zero denominator is neither 100% nor 0% (NOT_APPLICABLE/UNVERIFIABLE). The oldcaught/plannedstays printed aslegacy.scripts/object_model.py(object-id/1,object-evidence/1): per-function/method objects with stable keys; every evidence field isMEASURED,NOT_APPLICABLEorUNVERIFIABLEwith a coded reason. A Go cover block is attributed only if it lies completely inside one object, nothing is split proportionally; build-tag variants are separate objects; repeated profiles markNONDETERMINISTIC. Rust objects come from the realcargo llvm-covexport, named with the pinned image'sc++filt; instances (generics, closures, per-binary copies) fold into one source object. The coverage kind isstatement,lineorregion: there is nobranch.NOT_CHECKEDwithout them), L3 completeness against an independent enumeration (PARTIALwhere only a part can be shown). There is no combined "verified"..measurement-scope.json(measurement-scope/1): every tracked Go/Rust file is classified, each exclusion has a reason, an unclassified file is an error;tools/goastdumpis measured and reported on its own.tools/objinventory: stdlib-only Go tool for exact function spans (a separate module, vetted and tested in the python job)..github/workflows/object-evidence.yml: report-only, not a required check, nothing needs it.Measured (pinned
rust:1.96.1-bookwormandgolang:1.26.10, this repo at 852594a)UNVERIFIABLE(8FILE_NOT_IN_PROFILE: theffi_cgobuild variants; 3NONDETERMINISTICover 6 runs).#[cfg(test)]item inffi/src/lib.rs); fixed with a tokenizing item-span detector. A consistently forged record (object removed, digest recomputed) passes L1 and fails L3.cargo llvm-cov --branchfails on the pinned stable toolchain (1 nightly option were parsed).teeth); the policy tests fail on the previous tree for the stated cause (7 behaviour-level failures, 6 structural ones that are not evidence).Not measured
object-evidence; until it is green on this head, the CI result is not measured. A draft PR from a deliberately broken branch (one production file left out of the inventory, so L3 must fail) is the CI proof of the red path; it is not to be merged.UNVERIFIABLE(MEASUREMENT_MISSING): the verdicts need a FULL run. Go branch coverage tooling, nightly branch coverage, function-level transitive impact and per-test attribution beyond a sample are not measured.Report and raw measurements: cic-factory
jobs/relay-ci-parity-04-object-level-measurement/output/.The Relay repository allows merge commits only.