Test262: run unflagged tests as sloppy+strict variant pairs (split from #882)

[nickna]

Summary

Test262's canonical harness runs each unflagged test (no flags: [onlyStrict] / [noStrict]) as two variants — once in sloppy mode and once with a "use strict" prologue. SharpTS currently runs each unflagged test once, sloppy only. This issue tracks adopting variant-pair execution so we match the canonical harness coverage.

Split out from #882 (now closed), where the investigation and the codegen prerequisites were completed. This is the remaining deliberate, maintainer-gated piece — a coverage/infrastructure enhancement, not a bug.

Why it was deferred from #882

Running unflagged tests as both variants is a baseline-semantics + coverage change, not a drop-in. Measured impact during #882:

• Baseline model changes — every unflagged test gains a second path → bucket entry (a strict variant). The current model is one result per path; variant-pairs make it two.
• Run time roughly doubles — every unflagged test executes twice.
• It surfaces a real strict-failure surface — a 495-test unflagged sample run force-strict initially showed ~9% (49/495) pass sloppy but fail strict, with 0 strict-only gains.

That made it a planned rollout decision rather than an autonomous change, consistent with #882's "regenerate the baseline only after triage" stance.

What's already done (prerequisite work, shipped in #884)

The concrete strict-mode codegen bugs that variant-pair coverage would expose were fixed proactively in #882/#884, so the runner change won't land on a pile of red. The 495-test force-strict sample went 49 → 40 → 34 → 27 → 8 → 4 → 0 sloppy-Pass→strict-Fail across the sweep:

• strict dynamic property writes on built-in objects persist
• strict writes on Date/RegExp/Promise/Error (PDS) objects persist
• strict delete removes Object.defineProperty (PDS) properties + honors configurability
• strict writes honor non-writable / accessor descriptors (TypeError / setter invocation)
• strict indexed writes on $Object receivers persist + honor preventExtensions
• strict symbol-keyed and globalThis-sentinel writes persist
• onlyStrict tests now actually run strict (Assemble() prepends a program-level "use strict")

So the codegen floor is in place; the sampled systematic strict clusters are resolved. A full (non-sampled) regen is still needed to confirm suite-wide before/at rollout.

Scope of this issue

1. Harness runner — for each unflagged test, execute both a sloppy variant and a "use strict"-prefixed strict variant (the strict prologue mechanism already exists from the onlyStrict fix). onlyStrict / noStrict-flagged tests keep running their single designated variant.
2. Baseline model — extend path → bucket to distinguish the two variants per unflagged path (e.g. a variant suffix/qualifier on the key). Applies to both baselines/compiled.txt and baselines/interpreted.txt, interpreter + compiled runners.
3. Full regeneration — regenerate both baselines under the new model (SHARPTS_TEST262_UPDATE_BASELINE=1) and record the resulting strict-variant Fail entries; triage any clusters a full run surfaces beyond the 495-test sample.
4. Run-time budget — confirm the ~2× wall-clock is acceptable for the subset config / CI, or gate strict variants behind a config flag if not.

Open decisions for the maintainer

• Go / no-go on adopting variant-pairs at all (vs. keeping sloppy-only and relying on onlyStrict-flagged tests for strict coverage).
• Baseline key encoding for the second variant.
• Default-on vs. opt-in flag, given the run-time cost.

References

• Parent: Test262: 62 deterministic compiled-mode regressions vs stale baseline (14 compiled-only codegen bugs) #882 (closed) — full investigation in the issue thread; the variant-pair gap is described in the "strict-mode conformance: runner + codegen; variant-pair gap scoped" status update.
• PR Fix #882: compiled-mode codegen regressions + strict-mode write/delete sweep + baseline regen #884 — the codegen prerequisites + baseline regen that this enhancement builds on.
• Harness: SharpTS.Test262/ (HarnessAssembler / Assemble(), baseline diff harness, baselines/{compiled,interpreted}.txt).

[nickna]

Full-subset scoping: measured strict-variant failure surface

Ran the deferred preview the issue calls for — the full (non-sampled) strict surface on the default config/subset.json. This quantifies exactly how much red variant-pairs would create at turn-on, so the go/no-go and rollout shape can be decided against real numbers rather than the 495-test extrapolation.

Method

Throwaway probe (reverted after measuring; no production change): env-gated Assemble() to force a "use strict" prologue on every unflagged test (noStrict/module excluded, raw returns early, onlyStrict already strict), then ran each mode strict-only with SHARPTS_TEST262_UPDATE_BASELINE=1 and diffed the result against the committed (sloppy) baselines. The Pass → Fail/RuntimeError transitions are precisely the new strict-variant entries variant-pairs would add. Verified zero worker-crashed/worker-stalled artifacts in either result set, so the data is clean. (The testhost did hit the pre-existing #882 Internal CLR error 0x80131506 during teardown — after both baselines were fully written, so results are intact.)

For reference: the default subset is 11,384 test files, 99.2% unflagged, so the variant-pair run-time cost lands at ~1.92× (only ~8% skipped tests keep it under a clean 2×).

Headline numbers

Mode	New strict-variant failures (Pass → fail)	"New passes" (strict-only gains)	Soft bucket shifts
Interpreted	13	41	38
Compiled	84	8	19

~97 tests go red at turn-on. The interpreter (13) is trivial (it already honors a program-level "use strict"); compiled (84) is the real surface, confirming the issue's codegen-centric framing. The "new passes" are tests already baselined Fail that happen to pass strict — not new work; under variant-pairs they become small new green.

The 84 compiled failures are not 84 bugs — ~5 root causes

By folder: Object 34, RegExp 21, Array 10, Number 8, Math 8, String 2, Error 1. By kind: Fail 72, RuntimeError 12. They cluster:

1. Strict assignment to a non-writable property must throw TypeError — compiled currently no-ops (~50 of 84). Confirmed by reading e.g. Math/PI/prop-desc.js = verifyProperty(Math, 'PI', {writable:false,...}), whose helper writes and expects a throw. Covers all Math/* (8), Number/* (8), Object/{freeze,seal,preventExtensions} (~21), and the scattered _A3 / descriptor.js / property-descriptor.js verifyProperty tests. This is the exact class Fix #882: compiled-mode codegen regressions + strict-mode write/delete sweep + baseline regen #884 fixed for $Object and PDS receivers — it just wasn't extended to built-in namespace objects (Math, Number) or to freeze/seal/preventExtensions enforcement in compiled. One coherent change clears the bulk.
2. RegExp lastIndex strict-write path (21). r.lastIndex = 1; r.exec(...) passes sloppy, fails strict — a compiled $RegExp write-path gap. Fix #882: compiled-mode codegen regressions + strict-mode write/delete sweep + baseline regen #884 fixed RegExp PDS writes against the 495-sample but missed lastIndex. ~1–2 fixes. This is the clearest example of the "sample-driven fixes left clusters" risk the issue anticipated.
3. Array length/index strict define throws the wrong exception type → RuntimeError instead of a catchable guest TypeError (7). 1 fix.
4. Stragglers (Object.is, assign/assign-descriptor, setPrototypeOf/property-descriptor) that likely fold into Great idea #1.

The interpreter's 13 are ~3 fixes: Array predicate call-with-boolean strict-this (4), JSON.stringify replacer/circular/bigint under strict (5), Object.defineProperties/defineProperty edge cases (4).

Net: ~97 red ≈ roughly a dozen actual fixes, all in the strict property-write family #884 already started. Contained, not a quagmire.

Recommendation: separate the infra decision from the fix work

The two scopes are independent and the harness already stores Fail entries:

• Infra (cheap, do now): runner double-execution + variant-keyed baseline + ~1.92× run time. You do not need to fix the 97 first — enable variant-pairs and baseline the strict failures as known-Fail. That turns on the coverage and the regression guard immediately (no new strict breakage can slip in), and leaves the 97 as a visible, trackable burn-down list.
• Fix work (bounded, incremental): clear the ~dozen root causes cluster-by-cluster. Cluster Great idea #1 (~50 tests, strict-write-throws on Math/Number namespace objects + freeze/seal/preventExtensions in compiled) is the single highest-leverage PR; Fix #882: compiled-mode codegen regressions + strict-mode write/delete sweep + baseline regen #884's machinery already exists to extend.

Gating strict variants behind an opt-in flag (or folding them into the existing wide-sweep config) keeps the default few-minute subset at 1× and avoids the ~1.92× hitting every CI run (relevant given the macOS 10-min timeout flake).