Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/falsify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,13 @@ jobs:
# Each mutation is a fresh copy plus a package build and one test run. The
# first copy pays for the whole module's compilation; the rest hit the cache.
#
# 150, not 30, and the figure is measured rather than chosen. The suite was
# 210, and each raise is measured rather than chosen. 30 was what cancelled
# this job for twenty-four consecutive nights (#737). 150 was set from a
# local projection of 97 minutes; the first night that actually finished
# took **145** (04:30 to 06:55 on 2026-09-20), leaving five minutes of
# margin on a suite that grew from 199 to 207 specs in the same week. A
# budget that tight turns the next dozen specs into a cancelled night, which
# is the failure this line already caused once. The suite was
# killed by this very line for twenty-four consecutive nights (#737): 30
# minutes was set when the suite was smaller, and it now carries 1271
# mutations across 199 specs. A local replay on a warm cache was clocked at
Expand All @@ -51,7 +57,7 @@ jobs:
# cancellation as an infrastructure finding rather than as a guard breaking.
#
# When this line needs raising again, split the specs across a matrix instead.
timeout-minutes: 150
timeout-minutes: 210
permissions:
contents: read
steps:
Expand Down
24 changes: 24 additions & 0 deletions internal/cli/evidence_earner_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,7 @@ func TestNoClientBorneAxisIsEarnedWithoutAClient(t *testing.T) {
}

witnessed := 0
var unwitnessed []string
for _, a := range evidenceAxisList() {
got := earnedWithoutAClient[a.Name]
if a.earner == earnedByAClient {
Expand All @@ -200,6 +201,29 @@ func TestNoClientBorneAxisIsEarnedWithoutAClient(t *testing.T) {
continue
}
witnessed += len(got)
if len(got) == 0 {
unwitnessed = append(unwitnessed, a.Name)
}
}
// Per axis, not only in total, and the difference is what #781 cost.
//
// The guard below is global: one witnessed axis satisfies it, and a
// declaration flipping a DIFFERENT axis to client-borne then passes
// unchanged. That is exactly what happened to `shape` — on 2026-09-21 the
// committed record held 21 undriven operations and not one of them carried a
// recorded answer, so the falsification for that axis proved nothing and a
// nightly replay read it as a guard that had stopped working.
//
// Logged rather than failed, on purpose. An axis no undriven operation earns
// is an ordinary state of the record, not a defect, and failing on it would
// be a red nobody can clear. What is not ordinary is nobody noticing, so the
// names are printed and a falsification can be aimed at an axis that still
// has a population.
if len(unwitnessed) > 0 {
sort.Strings(unwitnessed)
t.Logf("this record cannot contradict a declaration on %d axis/axes, because no undriven "+
"operation earns them: %s. A falsification aimed at one of those proves nothing.",
len(unwitnessed), strings.Join(unwitnessed, ", "))
}
if witnessed == 0 {
t.Fatal("no axis declared earnable without a client was earned by any undriven operation, " +
Expand Down
25 changes: 21 additions & 4 deletions internal/core/emulator/attribution_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,20 +26,37 @@ import (
// they touch the store. Nothing about the old attribution could survive it: two
// non-probe requests were in flight for every touch, so every touch was
// dropped.
//
// Two barriers, not one, and the second is what makes the falsification
// deterministic. With only the entry barrier, both handlers started together and
// then raced: a handler that finished its touches and returned left the flight
// before the other touched anything, so the other attributed correctly even with
// the goroutine check removed. The mutation went undetected in roughly three
// runs out of five — measured on 2026-09-21 — which reads as "this guard stopped
// working" in a nightly replay and is nothing of the sort.
//
// The exit barrier holds both requests in flight until BOTH have touched, which
// is what the sentence above always claimed and what the guard actually has to
// survive.
func barrierPack(env *emulator.Env) stubPack {
const provider, kind = "stub", "thing"
tenant := resource.Tenant{Provider: provider}

// Both handlers meet here before the first store touch and leave together,
// so the overlap is a fact of the test rather than a hope about timing.
var barrier sync.WaitGroup
barrier.Add(2)
var entry, exit sync.WaitGroup
entry.Add(2)
exit.Add(2)
cycle := func(w http.ResponseWriter, _ *http.Request) {
barrier.Done()
barrier.Wait()
entry.Done()
entry.Wait()
id := env.NewID()
env.Store.Put(&resource.Resource{ID: id, Kind: kind, Tenant: tenant})
env.Store.Delete(provider, kind, id)
// Nobody leaves until both have touched, or the last toucher is alone in
// flight and any attribution rule at all would get it right.
exit.Done()
exit.Wait()
emulator.WriteJSON(w, http.StatusOK, map[string]string{"id": id})
}
return stubPack{name: provider, routes: []emulator.Route{
Expand Down
4 changes: 2 additions & 2 deletions tools/falsify/specs/a-silent-night-is-refused.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@
"test": "TestEveryScheduledWorkflowReportsOrSaysWhyNot"
},
{
"label": "the replay is given back the thirty minutes that killed it every night",
"label": "the replay is given back the thirty minutes that killed it every night, measured against a run that now needs 145 (#737, #781)",
"file": ".github/workflows/falsify.yml",
"package": "./tools/ci/",
"find": " timeout-minutes: 150",
"find": " timeout-minutes: 210",
"replace": " timeout-minutes: 30",
"test": "TestTheFalsifySuiteIsGivenTimeToFinish"
}
Expand Down
16 changes: 8 additions & 8 deletions tools/falsify/specs/probe-side-zeros.json
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
{
"package": "./internal/cli/",
"subject": "which axis a reason is allowed to explain, in the work queue (#445)",
"why": "`--gaps` filed a zero as `declared` \u2014 \"not work: no path exists to close this zero\" \u2014 as soon as the record said no client drove the operation, on all seven axes at once, and printed the route's Route.Undriven reason beside it. That reason is a sentence about clients. The contract-driven probe needs no client, so on `probed` and `contract` it explained nothing, and on `shape` \u2014 resolved offline from the recordings catalogue \u2014 it explained nothing either. Measured on the committed record: 38 queue lines across 22 operations said nobody could act, and #429 had just shown from the other side that 31 Scaleway operations earn `contract` and 29 earn `probed` from a single fix to the contract extraction, with no client and no pack code touched. Each mutation below puts one of those 38 lines back, or takes away a witness that keeps them out.",
"why": "`--gaps` filed a zero as `declared` — \"not work: no path exists to close this zero\" — as soon as the record said no client drove the operation, on all seven axes at once, and printed the route's Route.Undriven reason beside it. That reason is a sentence about clients. The contract-driven probe needs no client, so on `probed` and `contract` it explained nothing, and on `shape` — resolved offline from the recordings catalogue — it explained nothing either. Measured on the committed record: 38 queue lines across 22 operations said nobody could act, and #429 had just shown from the other side that 31 Scaleway operations earn `contract` and 29 earn `probed` from a single fix to the contract extraction, with no client and no pack code touched. Each mutation below puts one of those 38 lines back, or takes away a witness that keeps them out.",
"mutations": [
{
"file": "internal/cli/evidence_gaps.go",
"find": "\tif axis.earner == earnedByAClient && !ev.Driven {",
"replace": "\tif (axis.earner == earnedByAClient || true) && !ev.Driven {",
"expect": "the axis stops being asked, so a zero on `probed` is retired again by a sentence about what the `exo` CLI cannot compose \u2014 and the queue tells a reader that work #429 has already done twice is work nobody can do",
"expect": "the axis stops being asked, so a zero on `probed` is retired again by a sentence about what the `exo` CLI cannot compose — and the queue tells a reader that work #429 has already done twice is work nobody can do",
"test": "TestAClientShapedReasonNeverExplainsAProbeSideZero",
"label": "a client reason explains a probe-side zero again"
},
Expand All @@ -21,11 +21,11 @@
},
{
"file": "internal/cli/evidence_axes.go",
"find": "\t\t\tName: \"shape\",\n\t\t\tearner: earnedByARecording,",
"replace": "\t\t\tName: \"shape\",\n\t\t\tearner: earnedByAClient + earnedByARecording - earnedByARecording,",
"expect": "a missing recording of the real cloud is declared to be a client's business, and exoscale/v2.get-operation \u2014 observed, and driven by nobody \u2014 says it is not",
"find": "\t\t\tName: \"contract\",\n\t\t\tearner: earnedByValidation,",
"replace": "\t\t\tName: \"contract\",\n\t\t\tearner: earnedByAClient + earnedByValidation - earnedByValidation,",
"expect": "a missing recording of the real cloud is declared to be a client's business, and exoscale/v2.get-operation — observed, and driven by nobody — says it is not",
"test": "TestNoClientBorneAxisIsEarnedWithoutAClient",
"label": "a missing recording is declared client-borne"
"label": "an axis earned by validation is declared client-borne, on an axis the committed record can actually contradict: `shape` had zero undriven witnesses on 2026-09-21, so the old mutation proved nothing and read as a guard that stopped working (#781)"
},
{
"file": "internal/cli/evidence_gaps.go",
Expand All @@ -39,7 +39,7 @@
"file": "internal/cli/evidence_gaps.go",
"find": "\t\t\t\t\tReason: why,",
"replace": "\t\t\t\t\tReason: reasons[op] + why[:0],",
"expect": "the printed reason is fetched again instead of coming from the branch that used it, so a line the classifier did not retire on a client reason prints one anyway \u2014 the exact decoupling the defect lived in",
"expect": "the printed reason is fetched again instead of coming from the branch that used it, so a line the classifier did not retire on a client reason prints one anyway — the exact decoupling the defect lived in",
"test": "TestAClientShapedReasonNeverExplainsAProbeSideZero",
"label": "the printed reason stops coming from the branch that used it"
},
Expand All @@ -52,5 +52,5 @@
"label": "the observer stops telling a probe from a client"
}
],
"note": "The sixth mutation runs against internal/core/emulator, not because the declaration lives there but because the fact it declares does: the observer is what keeps a synthetic exchange apart from a client's, and every axis reads one side or the other. A declaration whose subject nobody drives is a comment, which is why TestOnlyAClientEarnsAClientBorneAxis issues one marked request and one plain one against a live emulator rather than reading the axis table back to itself. The second and third mutations are the other witness \u2014 the committed record \u2014 and they are kept separate on purpose: the mechanical one cannot see a mis-declaration of an axis the two exchanges happen not to move, and the record one cannot see a boundary that has stopped existing. The sixth mutation edits a file of internal/core/emulator and deliberately does NOT carry a `package` of its own: the test that reads it lives in internal/cli, and the first draft pointed the run at the emulator package, where `-run` matched nothing and the mutation came back green with no test having run at all. The harness refused the spec rather than certifying it, which is the behaviour to rely on \u2014 a mutation whose test never executes reads exactly like a guard that does not bite."
"note": "The sixth mutation runs against internal/core/emulator, not because the declaration lives there but because the fact it declares does: the observer is what keeps a synthetic exchange apart from a client's, and every axis reads one side or the other. A declaration whose subject nobody drives is a comment, which is why TestOnlyAClientEarnsAClientBorneAxis issues one marked request and one plain one against a live emulator rather than reading the axis table back to itself. The second and third mutations are the other witness — the committed record — and they are kept separate on purpose: the mechanical one cannot see a mis-declaration of an axis the two exchanges happen not to move, and the record one cannot see a boundary that has stopped existing. The sixth mutation edits a file of internal/core/emulator and deliberately does NOT carry a `package` of its own: the test that reads it lives in internal/cli, and the first draft pointed the run at the emulator package, where `-run` matched nothing and the mutation came back green with no test having run at all. The harness refused the spec rather than certifying it, which is the behaviour to rely on — a mutation whose test never executes reads exactly like a guard that does not bite."
}
Loading