Skip to content

test(stress): debug build for the soak server + per-interval throughput - #326

Merged
cryo2010 merged 1 commit into
mainfrom
test/stress-debug-build-and-throughput
Sep 24, 2026
Merged

cryo2010 merged 1 commit into
mainfrom
test/stress-debug-build-and-throughput

Conversation

@cryo2010

Copy link
Copy Markdown
Owner

What

Two changes to the per-workload stress soaks (nimble stress*), aimed at making the soaks better at their actual job: forcing and diagnosing intermittent crashes.

1. Build the soak server -d:release, not -d:danger

The soaks exist to force intermittent crashes and instability, but -d:danger disables all runtime checks (bounds/overflow/nil) plus assertions. That works against the goal: without checks, a defect doesn't fail at its site, it silently corrupts memory and crashes somewhere unrelated (the hardest kind to trace). It also makes --panics:on a no-op, since the Defects panics govern are never raised in the first place.

-d:release keeps optimizations on (realistic timing, so races/lifecycle bugs still reproduce) but keeps the checks on, so a defect fails fast at its source. Added --stackTrace:on --stackTraceMsgs:on --lineTrace:on --panics:on --debugger:native for a native traceback on failure.

Tradeoff: with checks on, the per-cell RSS/throughput are a debug-build baseline, not the peak ceiling. That's the correct tradeoff for a debugging build; the Dockerfile comment is updated to say so.

2. Per-interval throughput on every report line

Non-streaming workloads (requests, methods, ws, sse) previously reported only cumulative 2xx tallies, so a throughput dip or declining trend was invisible unless you diffed successive lines by eye. Added a per-interval 2xx completion rate (requests/s / messages/s / events/s per the workload), computed the same way fmt_xfer already does MB/s for the streaming workloads. 0 in the segment means nothing completed in the interval, i.e. a stall.

[requests h2 sync] 200x1481767 | 33018 requests/s | RSS 29MB | heap 7MB | fds 42 | t=45s
[requests h2 sync] final 200x1493782 | 802 requests/s | RSS 29MB | heap 6MB | fds 41 | t=60s

Notes

  • The rate's resolution is VORTEX_REPORT_SECONDS (default 60s). For dip-hunting, run with a smaller value (e.g. VORTEX_REPORT_SECONDS=5) for a finer time series.
  • The rate is interval throughput (delta/dt), not a running average, so a dip shows as a low number on that one line rather than being smoothed away.

Test

  • python3 -c "import ast; ast.parse(...)" on the client passes.
  • The changes are local-only stress tooling (Docker soaks), not a CI gate.

Build the stress soak server -d:release instead of -d:danger. The soaks exist
to force intermittent crashes and instability, and -d:danger disables the very
runtime checks (bounds/overflow/nil) that would fail a defect at its source; it
also makes --panics:on a no-op since the Defects it governs are never raised.
Keep optimizations on (realistic timing to reproduce races/lifecycle bugs) and
add --stackTrace/lineTrace/stackTraceMsgs/panics/debugger for a native
traceback. RSS/throughput become a debug-build baseline, not the peak ceiling.

Add a per-interval throughput rate to every non-streaming report line (2xx
completions/s; requests/messages/events per the workload). Cumulative code
tallies alone hide a dip or a declining trend, which is exactly what a soak
should surface; the streaming workloads already print MB/s via fmt_xfer. 0 in
the segment means nothing completed in the interval, i.e. a stall.
@cryo2010
cryo2010 merged commit 2a7adf8 into main Sep 24, 2026
35 checks passed
@cryo2010
cryo2010 deleted the test/stress-debug-build-and-throughput branch September 24, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant