Skip to content

fix: deterministic read and output order (worker-count independent) - #22

Merged
bnemoz merged 1 commit into
mainfrom
v2/deterministic-order
Nov 18, 2026
Merged

bnemoz merged 1 commit into
mainfrom
v2/deterministic-order

Conversation

@bnemoz

@bnemoz bnemoz commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Two runs of the same code on the same input gave different outputs. run() collected parse_barcodes chunk results and process_droplet results with as_completed, so read order within a well (and the row order of every output) depended on worker scheduling. Read order drives greedy-clustering tie-breaks and POA input order, so some equal-count clusters got a 1-nt different consensus or a different UMI count, and pair-pooled jittered by ±0.1 pt precision/recall and ±2 pairs. Benchmarks could not call neutral changes SAME.

What

  • run(): chunk results are concatenated in chunk (= input file) order, so reads reach clustering/POA in input order. Droplets are submitted in sorted barcode order and metadata is assembled in that order (progress bar still updates on completion). Annotated contigs are natural-sorted by sequence_id after abstar. pair_well writes pairs natural-sorted by barcode (both modes).
  • pair-pooled: explicit tie-breaks. Representative contig per (species, well): most UMIs, then smallest sequence_id. Rows are put in a fixed order before the home-posterior log-sum-exp, so float sums always run in the same order. Reciprocal-best ranking breaks equal confidence by well, then sequence_id:0, then sequence_id:1. Output sorted by confidence, then barcode/well/ids. Only ties are affected.
  • QC: (barcode, umi) is sorted before the seeded shuffle of the UMI-error null (unique() order was random). The failure table breaks ties by mode.
  • Lane C finding (a): early exits of a well (no valid barcode, empty metadata, ≤1 consensus, empty file) now delete temp chunks, close printers and advance the main bar. Before this, they left chunk files in temp/.
  • Policy: the fixed order applies in every mode, legacy=True included. There is no flag. v2-base's own order was random, so the fixed order is one valid v2-base outcome; the legacy docstring says so. The golden test passes unchanged: its wells are single-chunk, and the input order is kept.

Not changed (lane C finding (b)): the cluster_fraction denominator (partition_df.shape[0]) counts empty reads that greedy_clustering skips. Fixing it would change legacy metadata whenever empty reads exist (e.g. all-G inserts under lstrip("G")), so this PR only reports it.

Tests

  • New pairplex/tests/test_determinism.py: golden wells plus a 1499-read, 2-chunk well, run with PAIRPLEX_MAX_WORKERS 1 and 2. All files under metadata/consensus/annotated/pairs/qc are byte-identical, metadata rows are in natural order, and pair-pooled gives identical frames (also on a repeat). It fails on origin/main (20 files differ, pooled differs) and passes here.
  • pytest pairplex: 146 passed, 1 skipped (annotation calls: abstar version mismatch; this skip predates the PR).

Benchmark

D5-determinism-b vs D5-determinism-a (same commit b673d07, sims only)

D5-determinism-b vs D5-determinism-a

[SAME] sim_cal / per_well
   precision            0.624 ->      0.624
   cell_recall          0.148 ->      0.148
   emitted               1572 ->       1572
   n_conf0.9                — ->          —
   prec_conf0.9             — ->          —

[SAME] sim_cal / pooled
   precision            0.634 ->      0.634
   cell_recall          0.154 ->      0.154
   emitted               1595 ->       1595
   n_conf0.9                8 ->          8
   prec_conf0.9         1.000 ->      1.000

[SAME] sim_cal / pairs
   precision            0.624 ->      0.624
   cell_recall          0.148 ->      0.148
   emitted               1572 ->       1572
   n_conf0.9                0 ->          0
   prec_conf0.9             — ->          —

[SAME] sim_imp / per_well
   precision            0.947 ->      0.947
   cell_recall          0.782 ->      0.782
   emitted               5870 ->       5870
   n_conf0.9                — ->          —
   prec_conf0.9             — ->          —

[SAME] sim_imp / pooled
   precision            0.966 ->      0.966
   cell_recall          0.870 ->      0.870
   emitted               6371 ->       6371
   n_conf0.9             4342 ->       4342
   prec_conf0.9         0.987 ->      0.987

[SAME] sim_imp / pairs
   precision            0.947 ->      0.947
   cell_recall          0.782 ->      0.782
   emitted               5870 ->       5870
   n_conf0.9                0 ->          0
   prec_conf0.9             — ->          —

runtime: 2030s -> 2027s (PASS, limit 1.5x)
OVERALL: PASS  (a set passes on its own rule; every scored set must pass)

Every output file of the two runs is byte-identical (diff -rq over pp/), pooled parquet included. Only stage_run.json and pooled_pairs_run_info.json differ, through their timestamps.

D5-determinism-v1 vs main-91397e0 (all 4 datasets)

Sims:

[SAME] sim_cal / per_well
   precision            0.624 ->      0.624
   cell_recall          0.148 ->      0.148
   emitted               1572 ->       1572
   n_conf0.9                — ->          —
   prec_conf0.9             — ->          —

[SAME~] sim_cal / pooled
   precision            0.633 ->      0.634
   cell_recall          0.154 ->      0.154
   emitted               1595 ->       1595
   n_conf0.9                8 ->          8
   prec_conf0.9         1.000 ->      1.000

[SAME] sim_cal / pairs
   precision            0.624 ->      0.624
   cell_recall          0.148 ->      0.148
   emitted               1572 ->       1572
   n_conf0.9                0 ->          0
   prec_conf0.9             — ->          —

[SAME] sim_imp / per_well
   precision            0.947 ->      0.947
   cell_recall          0.782 ->      0.782
   emitted               5870 ->       5870
   n_conf0.9                — ->          —
   prec_conf0.9             — ->          —

[SAME~] sim_imp / pooled
   precision            0.966 ->      0.966
   cell_recall          0.871 ->      0.870
   emitted               6371 ->       6371
   n_conf0.9             4342 ->       4342
   prec_conf0.9         0.987 ->      0.987

[SAME] sim_imp / pairs
   precision            0.947 ->      0.947
   cell_recall          0.782 ->      0.782
   emitted               5870 ->       5870
   n_conf0.9                0 ->          0
   prec_conf0.9             — ->          —

Real data (per the public-repo rule: verdict and purity delta only):

set verdict clonal purity Δ
real_d02 / per_well PASS (SAME~) 0.0 pt
real_d02 / pooled PASS (SAME~) −0.1 pt
real_d02 / pairs PASS (SAME~) 0.0 pt
real_d07 / per_well PASS (SAME~) 0.0 pt
real_d07 / pooled PASS (SAME~) 0.0 pt
real_d07 / pairs PASS (SAME~) 0.0 pt

Lineages: PASS on all six real sets, 0 lost lineages at ≥3/5/10; cells recovered ≥0.976. Runtime 18884 s → 18793 s (PASS).
On real data, emitted pair counts move by a handful (≤0.01%). That is expected: main's read order was random, so its clustering/POA tie-breaks were one random draw, and this PR fixes a single order. Sim per-well/pairs are exactly SAME.

OVERALL: PASS

🤖 Generated with Claude Code

- run(): collect parse_barcodes chunk results in chunk order (not as_completed),
  so reads reach greedy clustering and POA in input-file order; process droplets
  in sorted barcode order and assemble metadata in that order; natural-sort
  annotated contigs by sequence_id; pair_well writes pairs naturally sorted by
  barcode. Applies in all modes, legacy included (documented in `legacy`).
- pair-pooled: explicit tie-breaks (representative contig: UMIs then
  sequence_id; reciprocal-best candidates: confidence, well, sequence_id:0/1),
  fixed row order before the posterior sums, deterministic output order.
- qc: sort (barcode, umi) before the seeded shuffle of the UMI null; failure
  table ties broken by mode.
- run(): early exits of a well (no valid barcode, no metadata, <=1 consensus,
  empty file) now clean up temp chunks, close printers and advance the bar.
- test_determinism: golden wells + a 2-chunk well, 1 vs 2 workers, byte-identical
  outputs and identical pair-pooled results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bnemoz
bnemoz marked this pull request as ready for review November 18, 2026 09:21
@bnemoz
bnemoz merged commit 8d50fab into main Nov 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant