Repository navigation
v2 lane C: support columns, per-stage drop counts, abstar failure table, QC report - #19
Merged
Merged
Conversation
Adds pairplex/umi.py:collapse_umis_directional (UMI-tools directional adjacency, neighbour generation, N-containing UMIs absorbed but never absorbing). process_droplet collapses UMIs per droplet partition: umi_raw keeps the raw UMI, metadata umis is the corrected count and umis_raw the raw count; min_cluster_umis filters on corrected counts. run(umi_collapse='directional'|'none'), CLI --umi-collapse; run(legacy=True) / --legacy forces 'none' (v2-base behaviour). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e, QC report
- annotated outputs carry cluster support: umis, reads, cluster_fraction,
barcode, contig (unpaired) and umis:0/1, reads:0/1, ... (paired); the
always-null abstar `umi` column is dropped (a consensus contig has no
single UMI; the count lives in `umis`)
- qc/stage_counts.csv: drop counts at every stage, per well and TOTAL
(reads, barcode groups, empty sequences skipped by greedy_clustering,
clusters failing each filter, annotation outcomes, chain composition,
emitted pairs)
- qc/annotation_failures.csv: abstar record errors (from its failures.tsv
warning), unassigned reasons, productivity issues, missing junctions
- qc/report.{json,md}: fragment fraction by locus, UMI Hamming-1 excess
vs shuffled null, heavy-junction multi-barcode share, leakage f by
chain (>= 8 wells), pair confidence distribution, clonal light-chain
purity with chance and a memory-only caveat
- new run() kwargs support_columns=True, qc=True, legacy=False (legacy
forces both off and reproduces v2-base); matching CLI options
- pairing is unchanged; logic lives in pairplex/qc.py behind small hooks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x/barcode_prefer_exact alongside umi_collapse; legacy forces all) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bnemoz
added a commit
that referenced
this pull request
Nov 18, 2026
- one legacy flag forcing every v2 option (now also pairing='exact') - qc chain-composition counts taken before the pairing rule; emitted_pairs from pair_well - umis:0/1 keep #19's meaning (the contig's own UMI count); the ranking support summed over merged same-clonotype contigs moves to chain_umis:0/1 (also in pairs/pairs.parquet) Stacked on #19 (which is stacked on #18): merge #18, then #19, then this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bnemoz
added a commit
that referenced
this pull request
Nov 18, 2026
legacy also forces pooling='per_well'; the per-well-only options (umi_collapse, pairing, min_dominance, collapse_clonal, support_columns, qc) are documented as not applying in cross-well mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PairPlex v2, lane C: carry support into the outputs, count drops at every stage, add a QC report. This covers fixes 9 and 10 (the PairPlex-side parts), plus "report drop counts at every stage", "count empty reads instead of skipping them silently" and the AbStar failure table, all from
Core fixes to do.md.Pairing is unchanged. All logic is in the new
pairplex/qc.py.run()gets only small hooks, so lane B (pairing block) and lane D (parser) can merge without conflicts.What changes
1. Support columns (
support_columns=True)umis,reads,cluster_fraction,barcodeandcontig.contigiscontig-N, where N is the rank by reads within the barcode. These are joined frommetadata/<well>.csvon the contig name, which is thesequence_id.umis:0/1,reads:0/1,cluster_fraction:0/1,barcode:0/1andcontig:0/1. These are the same objects, soassign_pairscarries them through.umi/umi:0/1are dropped instead of filled.umi_pattern. PairPlex never setsumi_pattern.umis, the name abutils already uses for 10x data.umi_count/consensus_count. This is easy to rename if preferred.Pair._chain_selectorsorts byumisonly when a barcode has several heavies or lights. The 1H1L rule removes those, so the emitted pairs stay the same.2.
qc/stage_counts.csv(per well and aTOTALrow)min_cluster_reads, and their reads;greedy_clusteringskips. These are counted only; behaviour is unchanged.assign_pairs, taken before the 1H1L rule: 1H1L, 1H0L, 0H1L, 1H2L, 2H1L, 2H2L, 0H0L, other.Everything is counted from frames that
run()already holds. Nothing is re-parsed.3.
qc/annotation_failures.csvColumns:
well, kind, mode, count, plusTOTALrows. On by default because it is cheap. Kinds:record_error: abstar reports these only through aRuntimeWarningthat points to itsfailures.tsv. PairPlex captures that warning, reads the file and then re-emits the warning. The mode isstage/category exception: message.unassigned: abstar'sfailure_reason.unproductive: one count perproductivity_issuesitem.no_junction.4.
qc/report.json+qc/report.md(written at the end ofrun())complete_vdjis reported next to it.pooled_pairing.estimate_leakageon the run's own per-well outputs when there are ≥8 wells.confidencecolumn exists, inpairs/pairs.parquetor in the paired outputs.D-duplicates are normalised, as in the harness.Flags
run()kwargs, documented in the docstring:support_columns=True,qc=True,legacy=False.legacy=Trueforces both new options off.--support_columns/--no_support_columns,--qc/--no_qc,--legacy.Local checks (2 simulated wells, 2 workers)
legacy=True: the paired/unpaired parquets are identical tov2-base, columns included.v2-baseon every shared column;umiwas removed and the support columns were added;clusters_pass == consensus_to_annotationandannotation_returned + failed_record_error == consensus_to_annotation.pairplex/tests/test_qc.pyhas 21 tests on small synthetic frames, one or more per counter and metric. The fullpairplexsuite gives 37 passed.Benchmark
C-support-qc-v1(default kwargs, all 4 datasets) vsv2-base: OVERALL PASS. Runtime was 17386 s -> 19116 s (1.10x).Simulated sets:
Real sets (clonal purity delta):
Lineage retention on the real sets:
Why not every set is SAME. v2-base is not deterministic from run to run, and this PR does not change pairing logic.
parse_barcodeschunks are collected withas_completed. The row order of the parsed reads in a well therefore changes between runs. I checked two local runs of v2-base code: same reads, different order.That order feeds greedy-clustering ties and POA input order. Between the two benchmark runs, 12/96 (sim_cal) and 28/96 (sim_imp) wells have a few clusters with the same read counts but a 1-nt different consensus or a different UMI count.
Those small consensus differences propagate to a few annotated contigs and so to pooled pairs. Per-well paired outputs are identical in all 192 sim wells.
This PR's hooks only read those frames: they never reorder or filter them.
A sims-only rerun of unmodified
v2-base(C-v2base-rerun-sim) measures this noise floor: the unmodified code shows the same pattern againstv2-base:The shifts seen here are therefore inside run-to-run noise.
Making the parse order deterministic (sort chunks, or sort reads by
seq_id) would make future benchmarks exact. That change belongs with lane D (parser).Notes for other lanes / upstream
complete_vdjisFalsefor every IGK contig on the sim (554/554), including contigs that start at germline position 2. The fragment metric therefore usesv_germline_startinstead./tmp/abstar-*diagnostics directory per call. That uses pod ephemeral storage on large runs.ValueError: CDR3 coordinates fall outside the assembled V(D)J sequenceand are dropped with no row. They are now visible inannotation_failures.csv.cluster_fractionuses the full partition size as its denominator, including the empty reads that clustering skips.run(), theparquet_chunks == []branchcontinues without updating the progress bar or cleaning up.🤖 Generated with Claude Code