Repository navigation
test(stress): debug build for the soak server + per-interval throughput - #326
Merged
Merged
Conversation
Build the stress soak server -d:release instead of -d:danger. The soaks exist to force intermittent crashes and instability, and -d:danger disables the very runtime checks (bounds/overflow/nil) that would fail a defect at its source; it also makes --panics:on a no-op since the Defects it governs are never raised. Keep optimizations on (realistic timing to reproduce races/lifecycle bugs) and add --stackTrace/lineTrace/stackTraceMsgs/panics/debugger for a native traceback. RSS/throughput become a debug-build baseline, not the peak ceiling. Add a per-interval throughput rate to every non-streaming report line (2xx completions/s; requests/messages/events per the workload). Cumulative code tallies alone hide a dip or a declining trend, which is exactly what a soak should surface; the streaming workloads already print MB/s via fmt_xfer. 0 in the segment means nothing completed in the interval, i.e. a stall.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two changes to the per-workload stress soaks (
nimble stress*), aimed at making the soaks better at their actual job: forcing and diagnosing intermittent crashes.1. Build the soak server
-d:release, not-d:dangerThe soaks exist to force intermittent crashes and instability, but
-d:dangerdisables all runtime checks (bounds/overflow/nil) plus assertions. That works against the goal: without checks, a defect doesn't fail at its site, it silently corrupts memory and crashes somewhere unrelated (the hardest kind to trace). It also makes--panics:ona no-op, since the Defects panics govern are never raised in the first place.-d:releasekeeps optimizations on (realistic timing, so races/lifecycle bugs still reproduce) but keeps the checks on, so a defect fails fast at its source. Added--stackTrace:on --stackTraceMsgs:on --lineTrace:on --panics:on --debugger:nativefor a native traceback on failure.Tradeoff: with checks on, the per-cell RSS/throughput are a debug-build baseline, not the peak ceiling. That's the correct tradeoff for a debugging build; the Dockerfile comment is updated to say so.
2. Per-interval throughput on every report line
Non-streaming workloads (
requests,methods,ws,sse) previously reported only cumulative 2xx tallies, so a throughput dip or declining trend was invisible unless you diffed successive lines by eye. Added a per-interval 2xx completion rate (requests/s/messages/s/events/sper the workload), computed the same wayfmt_xferalready does MB/s for the streaming workloads.0in the segment means nothing completed in the interval, i.e. a stall.Notes
VORTEX_REPORT_SECONDS(default 60s). For dip-hunting, run with a smaller value (e.g.VORTEX_REPORT_SECONDS=5) for a finer time series.Test
python3 -c "import ast; ast.parse(...)"on the client passes.