Skip to content

ci(go): run internal/server in its own test invocation, alone - #874

Merged
remyluslosius merged 2 commits into
mainfrom
ci/isolate-server-test-suite
Sep 21, 2026
Merged

remyluslosius merged 2 commits into
mainfrom
ci/isolate-server-test-suite

Conversation

@remyluslosius

Copy link
Copy Markdown
Contributor

Summary

CP bugs/OW-066. Founder-authorized for implementation and testing only; not for merge. Holds on #871 and #873 stay.

Problem. internal/server took 794 s on main a5a05056 under the 900 s per-package ceiling and timed out on two PRs with no failing test (the same tests running 27% to 48% slower at the median); a rerun of the same head then passed in 314 s. The variance's cause is unconfirmed; the margin is not.

Change, meeting the recorded conditions:

  1. Every package runs exactly once: phase 1 is every package except internal/server (go list ./... | grep -v on the exact package) under the shared -race -json -timeout 900s -p 4; phase 2 is internal/server alone. A package-count guard fails the step if the split stops closing.
  2. Other packages keep 900 s. This first revision keeps phase 2 at 900 s too, so the isolation effect is measured on its own before any budget change; the 1800 s revision follows as a second commit.
  3. Both exit statuses preserved: a green phase 2 never masks a red phase 1 (verified for all four combinations with a stub go).
  4. Both JSON streams (/tmp/go-test.json, /tmp/go-test-server.json) written on every outcome, both uploaded, both ingested by specter ingest --go-test ... --go-test ....
  5. Explicit timeout-minutes: 60 on the job (was the 360-minute platform default): two ceilings plus the measured 5 to 8 minutes for the rest.
  6. Validation plan: one run at this revision (isolation at 900 s), then three runs on the 1800 s revision, compared by the same artifact method against main and the failed attempts.

Contract. release-ci-gates 1.18.1 restates AC-09 for the two-invocation shape; its test pins the exclusion, the separate invocation with its own -timeout, both ingest paths, the second upload path and the explicit job budget (mutation checked). Workflows README updated.

Not changed

Per-test database creation (443 clones per run) is the durable fix and is tracked separately. No other workflow, code or contract changes.

internal/server is the database-heavy suite. On the hosted runner it
took 794 s on main a5a0505 and timed out at the 900 s per-package
ceiling on two PRs with no failing test: the same tests, running 27% to
48% slower at the median, while sharing the PostgreSQL service with
three sibling packages under -p 4. A rerun of the same head then passed
in 314 s. The cause of the variance is not confirmed; the margin is
(CP bugs/OW-066).

The test step is now two invocations inside one step, every package
exactly once: phase 1 runs every package except internal/server under
the shared 900 s ceiling, so a hang anywhere else still fails at 15
minutes; phase 2 runs internal/server alone. This commit keeps phase 2
at 900 s on purpose, to measure isolation on its own before any budget
change. Both exit statuses are preserved (a green phase 2 never masks a
red phase 1; verified for all four combinations with a stub go), both
JSON streams are written and uploaded on every outcome, specter ingests
both, and a package-count guard fails the step if the split ever stops
closing. The job declares timeout-minutes 60 instead of relying on the
360-minute platform default: two ceilings plus the measured 5 to 8
minutes of build, vet, lint, vuln, specter, vitest and upload.

release-ci-gates 1.18.1 restates AC-09 for the two-invocation shape and
its test pins the exclusion, the separate invocation with its own
-timeout, both ingest paths, the second upload path and the explicit
job budget; removing the second ingest path turns it red. The workflows
README describes the same. The detect-secrets baseline is the hook's own
line-number refresh.

CP: bugs/OW-066
The preliminary run (35635263290, revision e5f9303) isolated
internal/server at the old 900 s ceiling and passed at 885 s: 442 tests,
median per-test ratio 1.00 against the contended base run on main, so
isolation alone does not restore the margin. Across five hosted runs of
the same inventory the package has ranged from 314 s to more than 900 s.
The budget is the operative change.

The internal/server invocation now runs under 1800 s: base 794 s times
the worst observed slowdown 1.63 is 1294 s, with room for one more such
excursion. Every other package keeps 900 s, so a hang elsewhere still
fails at 15 minutes; a hang in internal/server fails at 30, accepted by
the founder as the cost. The job budget of 60 minutes already covers
both ceilings. Nothing else changes; this is the revision the three
validation runs measure.

CP: bugs/OW-066
@remyluslosius
remyluslosius merged commit 1520979 into main Sep 21, 2026
32 of 33 checks passed
@remyluslosius
remyluslosius deleted the ci/isolate-server-test-suite branch September 21, 2026 19:47
remyluslosius added a commit that referenced this pull request Sep 27, 2026
… guidance

CHANGELOG [Unreleased] gains the ten PRs merged after v0.8.0-rc.5 (#870
to #879; #874 is CI-only and is not listed). Upgrade notes lead: the
0065 migration signs everyone out, cookie logout requires the CSRF
token, and the audit export refuses an unknown parameter. Each entry was
checked against the merged code: the 0065 migration body, the binder's
sid check and EvaluateBearerBinding, the logout CSRF branch, the
LockWaitBound, OperationDeadline and RollbackCleanupLimit constants, and
the auth.login.failure and admin.user.enabled declarations in
audit/events.yaml. Known limitations name CP bugs/OW-072 and OW-062.

QUICKSTART's incident step said active sessions end "via logout" and
told operators to rotate passwords. Logout ends one login, and a user's
own password change signs out nothing else (OW-072). It now names
disable and the administrator reset, which end every interactive
credential since #875 and #876.

SECURITY_INCIDENT said an access token is ended only by rotating the
signing key. Since #876 it names its session and is refused once that
session is revoked, so revoking the rows ends it with no restart. Key
rotation is kept, scoped to a key that may itself be exposed.
remyluslosius added a commit that referenced this pull request Sep 27, 2026
… guidance (#883)

* docs(release): record the changes since rc.5 and correct the incident guidance

CHANGELOG [Unreleased] gains the ten PRs merged after v0.8.0-rc.5 (#870
to #879; #874 is CI-only and is not listed). Upgrade notes lead: the
0065 migration signs everyone out, cookie logout requires the CSRF
token, and the audit export refuses an unknown parameter. Each entry was
checked against the merged code: the 0065 migration body, the binder's
sid check and EvaluateBearerBinding, the logout CSRF branch, the
LockWaitBound, OperationDeadline and RollbackCleanupLimit constants, and
the auth.login.failure and admin.user.enabled declarations in
audit/events.yaml. Known limitations name CP bugs/OW-072 and OW-062.

QUICKSTART's incident step said active sessions end "via logout" and
told operators to rotate passwords. Logout ends one login, and a user's
own password change signs out nothing else (OW-072). It now names
disable and the administrator reset, which end every interactive
credential since #875 and #876.

SECURITY_INCIDENT said an access token is ended only by rotating the
signing key. Since #876 it names its session and is refused once that
session is revoked, so revoking the rows ends it with no restart. Key
rotation is kept, scoped to a key that may itself be exposed.

* docs(runbook): disable a compromised account instead of deleting it

Three defects in SECURITY_INCIDENT, all present in v0.8.0-rc.5 (CP
bugs/OW-082), kept in their own commit so they can be dropped
independently.

- "There is no is_active flag; disabling an account means
  soft-deleting it." POST /api/v1/users/{id}:disable has existed since
  #601 and, since #875, ends every interactive credential. The section
  now leads with disable, which :enable reverses, and keeps delete and
  the SQL fallback with what each does and does not do.
- The delete was said to be audited as account.user.deleted, which is
  the host-side /etc/passwd event. DeleteUserByID emits
  admin.user.deleted.
- Recovery verification step 3, headed "No live sessions for disabled
  accounts", checked only deleted_at. It now checks disabled_at too.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant