ci(nextest): print the output of a failed attempt that a retry then passes - #2520
Merged
Merged
Conversation
…asses With status-level = "fail", a test that fails and passes on retry reports only FLAKY 2/3, so the reason for the failed attempt is lost. Retries hid the batch/EC2/CodeBuild image-pull flakes for weeks this way. Show retried failures, with their output, so each flake names its cause the first time it happens.
This was referenced Sep 14, 2026
Sorttech
pushed a commit
to Sorttech/fakecloud
that referenced
this pull request
Sep 24, 2026
… time TestFunction and TestConnectionFunction ran the handler on a worker thread and failed it with "function execution exceeded the 250ms time limit" when the thread had not answered within 250ms of wall-clock time, counting thread spawn, runtime setup and any time the host spent not running the thread. On a loaded CI runner a trivial echo handler hit that limit: the cloudfront_test_function e2e tests needed a retry in 4 of the last 26 E2E runs, and nextest's retried-failure output (faiscadev#2520) captured the error on test_function_echoes_request. Real CloudFront Functions are bounded by compute (~1ms of CPU per request), not elapsed time. The worker now measures its own CPU time (CLOCK_THREAD_CPUTIME_ID; elapsed time where the platform has no thread CPU clock) and a run that used more than 250ms of it fails with the same error, keeping whatever it logged. ComputeUtilization is derived from that CPU time too. The caller's recv_timeout becomes a 10s wall-clock safety net for a run that never returns. boa's loop and recursion caps still stop `while(1){}`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
cinextest profile usesstatus-level = "fail", so a test that fails and then passes on retry leaves only aFLAKY 2/3line. The failed attempt's panic and output are never printed, so the flake can't be diagnosed from CI and just gets retried again.That is how the Batch, EC2 and CodeBuild image-pull flakes (#2516, #2519) went unexplained for weeks, and it is what blocks diagnosing the ones still open, e.g.
cloudfront_test_function::test_function_throw_populates_error_message(4 retries in the last 26 E2E runs, not reproducible locally).status-level = "retry"prints each failed attempt with its output (failure-output = "immediate-final"already applies), thenRETRY 2/3and the passing attempt. Runs with no failed attempts print exactly what they do today, so log volume only grows when something actually failed.Used by the E2E and Conformance workflows (
nextest run -P ci).Test plan
TRY 1 FAIL, the attempt's stderr including the panic message,RETRY 2/3,TRY 2 PASS, and theFLAKY 2/3summary. Withstatus-level = "fail"only theFLAKY 2/3summary line appears.Summary by cubic
Changes the CI nextest profile to use
status-level = "retry"so a test that fails but passes on retry prints the failed attempt's output instead of leaving only aFLAKY 2/3line. This makes flakes diagnosable from CI logs without needing to reproduce them locally. Runs with no failed attempts are unchanged; log volume only grows when something actually fails.Written for commit b4409ba. Summary will update on new commits.