The omission gate judges a run that was cut short, and blames the suites that never ran #789

The omission gate declares a whole run by the name of its leg, not by what the leg actually did. When a suite fails early and the rest are skipped, the gate still judges, and it blames whatever the skipped suites would have exercised.

Twice in ten days, both times with an unrelated external cause, and both times the printed verdict named Outscale.

Measured

2026-09-21, on #788

The Terraform provider could not be installed:

Error while installing scaleway/scaleway v2.83.0: could not query provider
registry for registry.terraform.io/scaleway/scaleway: failed to retrieve
cryptographic signature for provider: the request failed after 2 attempts,
please try again later: 500 Internal Server Error returned from github.com

Step 16 of 25 failed, and steps 17 through 25 were skipped: the Outscale and Exoscale Terraform suites, the example stacks, the quick start, fault injection, the declared catalogue, the recorded refusals, the declared environment, and the probe.

The step that then ran — Report what was exercised, if: always() — printed:

conformance score: 330 of 401 routes proven by a real client
FAIL: fields the real cloud returns and no answer of this run carried:
  osc/Client.CreateSecurityGroupRule: SecurityGroup.InboundRules[].IpRanges: array
  osc/Client.DeleteRoute: RouteTable.LinkRouteTables[].SubnetId: string
  osc/Client.ReadNics: Nics[].LinkNic: object, Nics[].LinkPublicIp: object
  osc/Client.ReadPublicIps: PublicIps[].NicId, PublicIps[].PrivateIp, PublicIps[].VmId
  osc/Client.ReadSecurityGroups: SecurityGroups[].InboundRules[].IpRanges: array
  osc/Client.ReadVms: Vms[].Nics: array, Vms[].PublicDnsName, Vms[].PublicIp
       Either serve them, or decline them in the pack's DeclinedFields() with a reason.

Six Outscale operations accused of omitting fields, by a run in which no Outscale Terraform suite had been allowed to start.

2026-09-11, on the metrics pull request

Same shape, different external cause:

Error while installing scaleway/scaleway v2.81.0: ...
read tcp 10.1.0.223:44282->185.199.110.133:443: read: connection reset by peer

Score fell to 329 of 397, and the same six Outscale operations were named. Green on the second attempt, with nothing changed.

Why it happens

.github/workflows/conformance.yml:

- name: Report what was exercised
  if: always()
  env:
    FEINT_FIELD_GATE: ${{ matrix.client == 'fields' && '1' || '0' }}
  run: tools/conformance/score.sh http://127.0.0.1:4599

FEINT_FIELD_GATE is 1 because the leg is called fields. It stays 1 whether that leg drove twenty-five steps or sixteen. And if: always() means the gate runs precisely on the runs where it should not.

tools/conformance/score.sh already states the principle it is being denied:

It judges a whole run, and only a whole run says so. […] So the gate is declared rather than assumed, the way a driver capability is […] An undeclared whole run counts as partial.

The declaration is sound. What is missing is that it is made before the work, as an intention, and never checked against the outcome. This is the distinction CLAUDE.md draws for the lifecycle:

L'état publié est celui que l'effet a produit, pas celui que l'intention visait.

And the one it draws for verdicts:

Un contrôle dont le verdict porte sur « cette exécution » doit être éprouvé sur l'exécution la plus pauvre qui le déclenchera, jamais sur la plus riche.

The poorest run that triggers this gate is not the probe leg, which #88 already fixed. It is the fields leg that died halfway, and nothing describes it.

Why it is worse than a flaky red

A flaky red is read as flaky. This one is read as a regression, and it names a provider nobody touched:

• the failing step is nine steps above the printed verdict;
• the verdict is precise, specific and plausible, listing real field names on real operations;
• the message ends with an instruction to act: "Either serve them, or decline them in the pack's DeclinedFields() with a reason." Following it would file six declines for fields the emulator serves correctly.

That last point is the real cost. The gate does not merely mislead, it invites a wrong change, and DeclinedFields() is exactly where a wrong entry survives for months because it reads as a decision somebody took.

What would close this

The gate should judge only what actually ran, and say so otherwise. The information exists: the job knows which steps were skipped, and /_feint/conformance knows which clients touched the emulator.

Sketch, not a prescription:

• the gate is armed by a fact rather than by a matrix name. The suites the fields leg is supposed to drive are declared somewhere both the workflow and score.sh can read, and the gate refuses to judge unless each of them reported in;
• refusing to judge prints the findings unjudged, the way a partial run already does today, and the run stays whatever colour the real failure made it;
• the non-vacuity guard that exists in internal/cli/evidence_earner_test.go is the model: it fails when the record cannot contradict the declaration, rather than passing on an empty population.

Whatever the shape, the property to hold is the one the gate's own comment already claims: a run that could not drive every suite is a partial run, whatever the leg is called.

Not in scope

The two provider-registry failures themselves. They are upstream outages, they are correctly reported as step failures, and a retry cleared both. What this issue is about is the verdict printed on top of them.