Skip to content

OMRF 10x lane: Cell Ranger / nf-core matrix → RustPipe-SC (not prairie-rna-stream) #3

Open

Description

@jwg054000

Tracking issue for the 10x Chromium lane of the OMRF DRAGEN replacement.

Relocated from prairie-rna-stream#15. That ticket is a scope fence: bulk RNA lives on prairie-rna-stream; 10x does not.

What this repo is

RustPipe-SC runs downstream of Cell Ranger / nf-core/scrnaseq. Inputs it already documents:

  • 10x Cell Ranger H5
  • H5AD (AnnData)
  • CSV (cells×genes or genes×cells)

QC → normalize → HVG → PCA → kNN → Leiden → markers. That is the Scanpy-speed step, not a Cell Ranger clone.

What this is not

  • Not prairie-rna-stream. Never feed a 10x / Cell Ranger BAM into STAR → rustqc as a bulk count matrix.
  • Not a Cell Ranger replacement (no BCL → FASTQ, no barcode/UMI counting here unless we later decide that is in scope).
  • Not DRAGEN DNA / onboard BCL. Those stay on the sequencer.

Honest next work (when we pick this up)

  1. Digest-pinned published image (same rule as bulk: do not vendor into prairie-rna-stream).
  2. First real 10x H5/H5AD through the pipeline with a packet contract the Prairie SDK can ingest.
  3. Fail-closed if someone hands us a bulk STAR BAM or rustqc featureCounts TSV.

Closing prairie-rna-stream#15 does not mean 10x is done. It means 10x is tracked here.

Activity

  1. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    PM board. This is the 10x lane. prairie-rna-stream stays bulk-only (stream#15 closed fence, stream#20 bulk human/CGC).

    Facts

    • Dockerfile for rustpipe-sc is already on main (PR Add rustpipe-sc Dockerfile for GHCR #2). GHCR package does not exist yet (ghcr.io/jwg054000/rustpipe-sc 404). .github/workflows/ is empty — no publish action.
    • This crate is downstream of Cell Ranger / nf-core/scrnaseq (H5 / H5AD / CSV). Not a Cell Ranger replacement. Not BCL→FASTQ. Not barcode/UMI counting unless already in src.
    • Local overnight-smoke/ is ~181 MB counts + .venv. Do not commit it. Add to .gitignore if missing.

    Next (smallest PR)

    1. GHCR publish workflow (linux/amd64), digest-pin story in README (docker pull only; prairie-rna-stream must never vendor this).
    2. PACKET.md: inputs H5/H5AD, outputs, fail-closed on bulk STAR BAM / rustqc featureCounts TSV.
    3. Comment the digest here once the image exists.

    Human click if the workflow cannot push packages: confirm write:packages on the repo and public GHCR (same lesson as rustpipe — anonymous HEAD 401 is normal until token redeem).

    Do not touch prairie-rna-stream Nextflow, the GRCh38 STAR pin, or i-00fcd62f6b296a2a0.

  2. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    Progress on the smallest useful 10x slice (not a Cell Ranger clone, not prairie-rna-stream).

    PR: #4

    #3 item status
    Digest-pinned published image workflow + Dockerfile ready; image not published until merge. ghcr.io/jwg054000/rustpipe-sc is 404 today.
    Packet contract for Prairie SDK PACKET.md — H5/H5AD/CSV in; actual pipeline CSVs + pipeline_timings.json out. No hashed run.json yet.
    Fail-closed bulk STAR BAM / rustqc featureCounts in — src/input_guard.rs rejects those as --input. Cell Ranger BAM stays --qc-bam only.

    Honest scope: no BCL→FASTQ, no barcode/UMI counting.

    Human clicks after merge:

    gh workflow run publish-ghcr.yml --repo jwg054000/RustPipe-SC
    
    gh api --method POST -H "Accept: application/vnd.github+json" \
      /user/packages/container/rustpipe-sc/visibility \
      -f visibility=public

    Then pin ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:<digest from the job summary>. Pull that digest; never docker build this tree from prairie-rna-stream.

  3. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    PM accept of PR #4 (merged, CI cargo test green). Honest scope: downstream of Cell Ranger, fail-closed on STAR BAM / featureCounts TSV / MTX. PACKET.md does not invent a hashed run.json yet (pipeline_timings.json is provenance).

    This issue stays open until:

    1. GHCR publish job on main succeeds (publish-ghcr.yml should have fired on the merge).
    2. Josh makes ghcr.io/jwg054000/rustpipe-sc public (first GITHUB_TOKEN package is private):
      gh api --method POST -H "Accept: application/vnd.github+json" \
        /user/packages/container/rustpipe-sc/visibility \
        -f visibility=public
      
      UI: https://github.com/users/jwg054000/packages/container/package/rustpipe-sc
    3. Digest pinned in README as ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:<digest> — do not float latest.

    Do not add this image to prairie-rna-stream. Bulk remains stream#20.

  4. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    GHCR is up and already public (no extra visibility click).

    Pin (do not float latest):

    ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1d
    

    Publish job: https://github.com/jwg054000/RustPipe-SC/actions/runs/33648130096

    Remaining on this ticket: put that digest in README. Do not add the image to prairie-rna-stream. Red-team nits on bulk GEO DE/PCA are not this repo.

  5. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    Correction: GHCR public is standing, not a human click. Sorry for the noise.

    Digest pin PR: #5
    ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1d

    Anonymous HEAD 401 is not private. This image still does not go into prairie-rna-stream.

  6. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    PM accept of #5 (merged, CI green). Digest pin is on main. GHCR is public (anonymous token redeem 200). Do not open another visibility or pin PR.

    Caught: merging the pin retriggered publish-ghcr.yml (push to main). Floating tag 0.3.0 now points at sha256:2d210034dce6d396e4fc4dce90dd61abd6015187883cb660920463bc93d05ffa. README still pins the first image sha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1d (digest still pullable). name:tag@sha256: uses the digest. Do not chase the tag — another docs PR rebuilds again.

    If a bot takes one small infra PR: path-filter publish-ghcr.yml to Dockerfile / Cargo.*, not README. Optional: add overnight-smoke/ to .gitignore (still missing).

    Remaining on this ticket: first real Cell Ranger H5/H5AD through pipeline (not PBMC 3K README numbers). Do not add this image to prairie-rna-stream. Bulk stays stream#20.

  7. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    Path-filter merged (#6). Docs PRs will not rebuild the image. Pin stays 0.3.0@sha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1d — do not chase the floating 0.3.0 tag. GHCR is public. No visibility click.

  8. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    H5 smoke (pinned digest) — packet PASS, default QC FAIL. Leaving #3 open.

    Ran on AWS test box i-00fcd62f6b296a2a0 (/data/sc, did not terminate, did not touch the in-flight bulk STAR/nextflow job). Image is the README pin, not floating 0.3.0:

    ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1d
    

    RepoDigests = that sha256. Image id sha256:a1c3207f6ab585c630289b34dedc341ac694cf2fd5882a7c0ebb4dc9e121a06c.

    H5

    Preferred pbmc3k URL (cell-exp/1.1.0/pbmc3k/..._h5.h5) 403 from the box (with and without a browser UA). Used documented fallback:

    https://cf.10xgenomics.com/samples/cell-exp/3.0.0/pbmc_1k_v3/pbmc_1k_v3_filtered_feature_bc_matrix.h5
    Host: /data/sc/pbmc_1k_v3_filtered_feature_bc_matrix.h5 (5.2 MB, HDF5 magic). Input to pipeline was this 10x H5, not a STAR BAM.

    Default flags panic (why this is not a close)

    docker run --rm -v /data/sc:/data $PIN \
      pipeline --input /data/pbmc_1k_v3_filtered_feature_bc_matrix.h5 --output /data/sc_out
    

    Loaded 1222 cells × 33538 genes, then QC at defaults (min_genes=200, max_pct_mt=5) kept 21 cells. PCA then:

    [pca-scaled] 21x2000 dense matrix, 50 components
    thread 'main' panicked at src/pca.rs:798:46:
    index out of bounds: the len is 441 but the index is 441
    

    pct_counts_mt on this v3 filtered H5: median 10.3, p90 18.4; only 21/1222 cells pass both min_genes=200 and max_pct_mt=5. PCA does not clamp n_pcs to min(n_cells, n_genes).

    Packet PASS (README 10x flags)

    docker run --rm --pull=never -v /data/sc:/data $PIN \
      pipeline --input /data/pbmc_1k_v3_filtered_feature_bc_matrix.h5 \
               --output /data/sc_out --max-pct-mt 20.0
    

    --max-pct-mt 20.0 is what README already shows for 10x H5. --pull=never because Docker 25 on the box hung looking up floating tag 0.3.0 (now a different digest); name:tag@sha256: still uses the digest once the image is local.

    [pipeline] COMPLETE in 0.385s — QC 1114 cells / 15186 genes, 18 Leiden clusters, pipeline=rustpipe-sc. No libqc/ (no --qc-bam).

    Host output: /data/sc/sc_out/

    file bytes PACKET columns
    qc_metrics.csv 50862 barcode,n_genes_by_counts,total_counts,pct_counts_mt (1222 cells, pre-filter)
    hvg_genes.csv 14120 gene (2000)
    pca_scores.csv 551250 sample,PC1…PC50 (1114 rows)
    knn.csv 225314 neighbor_1,distance_1… k=15
    clusters.csv 23707 barcode,cluster (1114 cells, clusters 0–17)
    markers.csv 20948 cluster,gene,score,pval,pval_adj,log2fc
    pipeline_timings.json 963 pipeline=rustpipe-sc, version=0.3.0

    pipeline_timings.json steps (seconds):

    step seconds
    load 0.0845
    qc_filter 0.1493
    normalize 0.0122
    hvg 0.0137
    scale 0.0041
    pca 0.0657
    knn 0.0098
    leiden 0.0059
    markers 0.0342
    total 0.3849

    Provenance: n_cells_raw=1222, n_cells_filtered=1114, n_genes_raw=33538, n_genes_filtered=15186, n_hvg=2000, n_pcs=50, knn_k=15, hvg_flavor=seurat, seed=42.

    Not done here

    • Did not close this issue (default pipeline --input/--output still panics on a real 10x filtered H5; PCA needs to clamp n_pcs).
    • Did not add this image to prairie-rna-stream. Did not vendor. Did not commit the H5 or /data/sc.
    • Did not chase floating tag 0.3.0.
    • Next useful crate fix: clamp PCA n_pcs (and fail-closed with a message if QC leaves too few cells), rather than another docs PR.
  9. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    PM: H5 smoke produced PACKET.md files with --max-pct-mt 20. Default flags panicked (21 cells, 50 PCs). Clamp merged as #7. GHCR will mint a new digest on src/ — pin that, do not float 0.3.0. Until the new image exists, keep using sha256:0f7164c5… + --max-pct-mt 20 for this H5. No --qc-bam. No Cell Ranger wrapper. prairie-rna-stream untouched.

  10. jwg054000 commented on Sep 2, 2026

    @jwg054000
    OwnerAuthor

    PM: Clamp image exists. Do not float 0.3.0.

    #7 publish succeeded: https://github.com/jwg054000/RustPipe-SC/actions/runs/33659706126

    ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:c1d9cc7581ac521e1dae5aae6ac2a63d457c822c4ddcc8507e457d44a21dc8c2
    

    (sha-94f94d1… and floating 0.3.0 both resolve to that digest.) README still pins sha256:0f7164c5…. SC bot: pin c1d9cc75… and re-smoke default flags on the same pbmc_1k_v3 H5 (that is the clamp test). Packet PASS on 0f7164c5 + --max-pct-mt 20 already stands. No --qc-bam. Do not add this image to prairie-rna-stream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions