Skip to content

Fix V(D)J boundary consistency, unify public output (0.9.0 breaking), add corpus CI - #135

Merged
briney merged 14 commits into
masterfrom
fix/vdj-region-mask-consistency
Nov 5, 2026
Merged

briney merged 14 commits into
masterfrom
fix/vdj-region-mask-consistency

Conversation

@briney

@briney briney commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

Incomplete junction anchors and region boundaries could abort annotation or produce inconsistent V(D)J region assemblies and masks. Recover boundaries conservatively from retained alignment evidence, partition amino-acid regions in the continuous query frame, and synchronize the FWR3/CDR3 endpoint. Ambiguous recovery remains an explicit record failure.

Per-sequence annotation errors now preserve diagnostics and a failure index while other records continue; strict=True / --strict restores abort-on-record-error behavior. Add focused biological regressions, native-Parquet auditing and exact run comparison tools, and docs/EDGE-CASES.md with grouped handling procedures and examples.

Replace the nightly corpus workflow with one push/PR job using a committed 50,000-sequence human BCR cohort: 40,000 stratified background records and 10,000 difficult records. Authenticate source identities and provenance, enforce conservation and independent consistency checks, and compare all 168 public output fields exactly against the reviewed regression baseline. Preserve duplicate IDs and require one individually documented ambiguous-boundary failure. Retain reports and diagnostics as workflow artifacts, document assignment-drift investigation and baseline review, and exclude the corpus from Python distributions.

Validation:

  • Full statement/branch suite: 1,678 passed; 91.25% coverage. Package and critical-module floors pass and were raised where supported by measurements.
  • Final focused runner, selection, workflow, and coverage-contract checks: 67 passed, including the additional imported-package provenance regression added after full-suite collection.
  • Three full 50,000-record runs agreed exactly: 49,999 annotated records, one expected failure, zero inconsistencies. The final check with two-core CPU affinity took 147.74 seconds.
  • Sphinx warning-as-error build and whitespace checks passed. Wheel and sdist built with installed build tools; archive inspection confirmed corpus exclusion.

The five-minute CI target still needs a hosted GitHub measurement; local timing excludes dependency installation. The frozen corpus baseline establishes regression behavior, not independent biological truth. The earlier full-source-corpus recovery investigation and its five separate assignment-reproducibility differences are documented in docs/superpowers/reports/2026-09-07-junction-anchor-validation.md.

Unified public output (breaking change for 0.9.0)

Every public output (AIRR TSV, Parquet, and the dataframe or Sequence objects returned by abstar.run()) now has the same fields, in the same order, with the same meanings, built by one conversion (abstar.annotation.airr.to_public_row) that runs once per record in the annotation workers. Writers only encode.

  • Coordinates are 1-based closed everywhere. Every *_start is one greater than in 0.8.x; every *_end is unchanged.
  • sequence is the original input everywhere, including returned Sequence objects. The assembled V(D)J (no constant region) and its translation are the new sequence_vdj / sequence_vdj_aa fields, used for every locus; the masks index them.
  • The AIRR-required fields lead the field order in every output.
  • A record whose coordinates cannot form a valid interval now fails individually in every format.
  • scripts/audit_annotation_consistency.py uses 1-based coordinates, checks sequence_vdj, and rejects 0.8.x Parquet.
  • Docs: a breaking-change warning with a before/after record in output_formats.rst, and a new changelog page (0.9.0, unreleased). version.py is not bumped.

Also fixes the corpus workflow, which GitHub had rejected because it used the runner context in job-level env; the corpus job now runs (about 5m22s on hosted runners).

Validation for this part:

  • Full suite: 1,689 passed, 4 skipped; 91.42% statement/branch coverage; all floors met. Sphinx warning-as-error build passed.
  • Corpus baseline regenerated and compared record by record with the previous one across all 49,999 annotated records. The only differences are the two new columns (sequence_vdj equals the oriented 1-based V(D)J slice) and every *_start shifted by exactly one. A full corpus check then passed with zero differences.
  • An independent whole-branch review found no critical issues; its two important findings are fixed.

🤖 Generated with Claude Code

briney and others added 11 commits September 7, 2026 11:53
GitHub rejects workflows that reference the runner context in job-level
env, so the corpus job never ran. Derive CORPUS_OUTPUT and MPLCONFIGDIR
from RUNNER_TEMP in a step and guard against the pattern in tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add sequence_vdj and sequence_vdj_aa, make OUTPUT_SCHEMA the single public
field order (AIRR-required fields first), and replace to_airr_row with
to_public_row, which maps every schema coordinate pair to 1-based closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Annotation workers now build each record's public row once, so work
Parquets, dataframe and Sequence returns, final Parquet, and AIRR TSV all
read the same values. Writers only encode; the per-row Parquet conversion
loop is gone. A row whose conversion fails becomes an isolated record
failure. Tests now assert public meanings: 1-based closed coordinates,
sequence as the input, and the V(D)J in sequence_vdj/sequence_vdj_aa.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The consistency audit now slices and checks region adjacency with
1-based closed coordinates, matching 0.9 public Parquet. The corpus
builder notes that its source Parquets predate the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recorded with --record-baseline and reviewed against the previous baseline
for all 49,999 annotated records: the only differences are the new
sequence_vdj/sequence_vdj_aa columns (sequence_vdj equals the oriented
1-based V(D)J slice) and every *_start shifted by exactly one. A full
corpus check then passed with zero differences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a 0.9.0 breaking-change warning with a before/after record to the
output-format guide, a new changelog page, and the sequence_vdj fields.
Python API, README, AGENTS, and edge-case docs now state that every output
shares 1-based closed coordinates and AIRR sequence meanings, with
Antibody internals the only 0-based representation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The audit now requires sequence_vdj, which only 0.9+ Parquet carries, so
0.8.x files with 0-based coordinates fail clearly instead of flagging
every record; it also checks sequence_vdj against the oriented slice.
The changelog notes this and that invalid-interval records now fail
individually in every output format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@briney briney changed the title Fix V(D)J boundary consistency and add committed corpus regression CI Fix V(D)J boundary consistency, unify public output (0.9.0 breaking), add corpus CI Nov 5, 2026
baltic 1.0 turned baltic.leaf into a module. abutils 0.6 uses it in a
typing.Union annotation, which Python 3.10 rejects at import time, so
`import abstar` fails on 3.10 before any test runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
briney and others added 2 commits November 5, 2026 10:48
abutils 0.6.1 constrains baltic<1 itself, so abstar no longer needs its
own baltic stopgap. Drop the arbitrary <0.7 cap on abutils, and move the
lowest-dependency CI profile and the corpus profile to abutils 0.6.1.

The corpus check with abutils 0.6.1 passed with no differences from the
baseline (49,999 annotated, 1 expected failure, 0 inconsistent).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The <2 and <26 caps excluded unreviewed releases rather than a known
incompatibility. With polars 2.1.0 and pyarrow 26.0.1 on Python 3.12, and
polars 2.0.0 and pyarrow 25.0.1 on Python 3.10, the full suite passes
(1689 passed, 4 skipped), and the corpus check matches the baseline
(49,999 annotated, 1 expected failure, 0 inconsistent).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@briney
briney merged commit 4475e18 into master Nov 5, 2026
30 checks passed
@briney
briney deleted the fix/vdj-region-mask-consistency branch November 5, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant