test(determinism): lock output against map iteration order (#210) - #242
Open
BenjaminDEMAILLE wants to merge 1 commit into
Open
test(determinism): lock output against map iteration order (#210)#242BenjaminDEMAILLE wants to merge 1 commit into
BenjaminDEMAILLE wants to merge 1 commit into
Conversation
4 tasks
…the SJ sorts total Junction counts live in a DashMap, whose iteration order varies with hashing and with concurrent insertion. Every path that emits an order already sorted, so no order was escaping, but two of those sorts keyed on (chr, start, end) while the key also carries strand and motif: a tie would have fallen back to the map's order. They now sort on the whole key. tests/determinism.rs is the lock: the same reads at one thread and at eight, and two runs at eight threads, must produce byte-identical SJ.out.tab and Aligned.out.sam, in single-pass and in two-pass mode, including pass 1's own SJ.out.tab. The test was checked against a positive control before being trusted: with the SJ writer ordering rows by a per-process random hash, all three tests fail. A deterministic permutation does not fail them, which is correct, and is why the control had to be per-process random rather than a fixed swap. Answers #210.
BenjaminDEMAILLE
force-pushed
the
audit/sj-iteration-order
branch
from
August 27, 2026 20:01
8352542 to
4623c0e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answers #210.
The trace
Every consumer of
SpliceJunctionStats::iter()and ofSpliceJunctionDb's map, with the verdict for each:compute_surviving_junctions(sj_output.rs:149)HashSet; the neighbour-distance pass reads the sorted vectorwrite_sj_lines(sj_output.rs:291)SJ.out.tabsj_feature_order(sj_output.rs:274)SJmatrix row positionsfilter_novel_junctions(junction/mod.rs:166)Vecis consumed only byinsert_novel, which inserts into aHashMap; contents, not orderinsert_novel→ two-pass DBsjdb_insert::build_gsjsort_and_dedup, and the junctions come from the GTF in file order, not from a mapSo no order was escaping. But two of those sorts keyed on
(chr, start, end)whileSjKeyalso carries strand and motif, which left a tie to fall back onDashMaporder. This PR sorts on the whole key in all three places, which is the remedy the issue itself proposes.Other maps, for the fourth checklist item:
DashMapappears nowhere else insrc/. The solo UMI paths (cellranger_1mm_map,directional,connected_components) already sort theirHashMapcontents on a total key (count, then UMI value) before the scan that depends on order.quantcounts into vectors indexed by gene, andgtfbuilds its gene list in file order.The lock
tests/determinism.rs: the same reads at one thread and at eight, and two runs at eight threads, must produce byte-identicalSJ.out.tabandAligned.out.sam, in single-pass and in two-pass mode, including pass 1's ownSJ.out.tab. The@PGheader is filtered because it records the command line, which contains the thread count.The control
A test that cannot fail proves nothing, so it was checked against one before being trusted. With the SJ writer ordering rows by a per-process random hash, all three tests fail. A deterministic permutation does not fail them — correctly, which is why the control had to be per-process random rather than the fixed swap I tried first.
cargo test,cargo clippy --all-targets -- -D warningson a cold cache, andcargo fmt --checkare green.🤖 Generated with Claude Code