You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PM board. This is the 10x lane. prairie-rna-stream stays bulk-only (stream#15 closed fence, stream#20 bulk human/CGC).
Facts
Dockerfile for rustpipe-sc is already on main (PR Add rustpipe-sc Dockerfile for GHCR #2). GHCR package does not exist yet (ghcr.io/jwg054000/rustpipe-sc 404). .github/workflows/ is empty — no publish action.
This crate is downstream of Cell Ranger / nf-core/scrnaseq (H5 / H5AD / CSV). Not a Cell Ranger replacement. Not BCL→FASTQ. Not barcode/UMI counting unless already in src.
Local overnight-smoke/ is ~181 MB counts + .venv. Do not commit it. Add to .gitignore if missing.
Next (smallest PR)
GHCR publish workflow (linux/amd64), digest-pin story in README (docker pull only; prairie-rna-stream must never vendor this).
PACKET.md: inputs H5/H5AD, outputs, fail-closed on bulk STAR BAM / rustqc featureCounts TSV.
Comment the digest here once the image exists.
Human click if the workflow cannot push packages: confirm write:packages on the repo and public GHCR (same lesson as rustpipe — anonymous HEAD 401 is normal until token redeem).
Do not touch prairie-rna-stream Nextflow, the GRCh38 STAR pin, or i-00fcd62f6b296a2a0.
workflow + Dockerfile ready; image not published until merge. ghcr.io/jwg054000/rustpipe-sc is 404 today.
Packet contract for Prairie SDK
PACKET.md — H5/H5AD/CSV in; actual pipeline CSVs + pipeline_timings.json out. No hashed run.json yet.
Fail-closed bulk STAR BAM / rustqc featureCounts
in — src/input_guard.rs rejects those as --input. Cell Ranger BAM stays --qc-bam only.
Honest scope: no BCL→FASTQ, no barcode/UMI counting.
Human clicks after merge:
gh workflow run publish-ghcr.yml --repo jwg054000/RustPipe-SC
gh api --method POST -H "Accept: application/vnd.github+json" \
/user/packages/container/rustpipe-sc/visibility \
-f visibility=public
Then pin ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:<digest from the job summary>. Pull that digest; never docker build this tree from prairie-rna-stream.
PM accept of PR #4 (merged, CI cargo test green). Honest scope: downstream of Cell Ranger, fail-closed on STAR BAM / featureCounts TSV / MTX. PACKET.md does not invent a hashed run.json yet (pipeline_timings.json is provenance).
This issue stays open until:
GHCR publish job on main succeeds (publish-ghcr.yml should have fired on the merge).
Josh makes ghcr.io/jwg054000/rustpipe-scpublic (first GITHUB_TOKEN package is private):
gh api --method POST -H "Accept: application/vnd.github+json" \
/user/packages/container/rustpipe-sc/visibility \
-f visibility=public
PM accept of #5 (merged, CI green). Digest pin is on main. GHCR is public (anonymous token redeem 200). Do not open another visibility or pin PR.
Caught: merging the pin retriggered publish-ghcr.yml (push to main). Floating tag 0.3.0 now points at sha256:2d210034dce6d396e4fc4dce90dd61abd6015187883cb660920463bc93d05ffa. README still pins the first image sha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1d (digest still pullable). name:tag@sha256: uses the digest. Do not chase the tag — another docs PR rebuilds again.
If a bot takes one small infra PR: path-filter publish-ghcr.yml to Dockerfile / Cargo.*, not README. Optional: add overnight-smoke/ to .gitignore (still missing).
Remaining on this ticket: first real Cell Ranger H5/H5AD through pipeline (not PBMC 3K README numbers). Do not add this image to prairie-rna-stream. Bulk stays stream#20.
Path-filter merged (#6). Docs PRs will not rebuild the image. Pin stays 0.3.0@sha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1d — do not chase the floating 0.3.0 tag. GHCR is public. No visibility click.
Ran on AWS test box i-00fcd62f6b296a2a0 (/data/sc, did not terminate, did not touch the in-flight bulk STAR/nextflow job). Image is the README pin, not floating 0.3.0:
RepoDigests = that sha256. Image id sha256:a1c3207f6ab585c630289b34dedc341ac694cf2fd5882a7c0ebb4dc9e121a06c.
H5
Preferred pbmc3k URL (cell-exp/1.1.0/pbmc3k/..._h5.h5) 403 from the box (with and without a browser UA). Used documented fallback:
https://cf.10xgenomics.com/samples/cell-exp/3.0.0/pbmc_1k_v3/pbmc_1k_v3_filtered_feature_bc_matrix.h5
Host: /data/sc/pbmc_1k_v3_filtered_feature_bc_matrix.h5 (5.2 MB, HDF5 magic). Input to pipeline was this 10x H5, not a STAR BAM.
Loaded 1222 cells × 33538 genes, then QC at defaults (min_genes=200, max_pct_mt=5) kept 21 cells. PCA then:
[pca-scaled] 21x2000 dense matrix, 50 components
thread 'main' panicked at src/pca.rs:798:46:
index out of bounds: the len is 441 but the index is 441
pct_counts_mt on this v3 filtered H5: median 10.3, p90 18.4; only 21/1222 cells pass both min_genes=200 and max_pct_mt=5. PCA does not clamp n_pcs to min(n_cells, n_genes).
--max-pct-mt 20.0 is what README already shows for 10x H5. --pull=never because Docker 25 on the box hung looking up floating tag 0.3.0 (now a different digest); name:tag@sha256: still uses the digest once the image is local.
[pipeline] COMPLETE in 0.385s — QC 1114 cells / 15186 genes, 18 Leiden clusters, pipeline=rustpipe-sc. No libqc/ (no --qc-bam).
PM: H5 smoke produced PACKET.md files with --max-pct-mt 20. Default flags panicked (21 cells, 50 PCs). Clamp merged as #7. GHCR will mint a new digest on src/ — pin that, do not float 0.3.0. Until the new image exists, keep using sha256:0f7164c5… + --max-pct-mt 20 for this H5. No --qc-bam. No Cell Ranger wrapper. prairie-rna-stream untouched.
(sha-94f94d1… and floating 0.3.0 both resolve to that digest.) README still pins sha256:0f7164c5…. SC bot: pin c1d9cc75… and re-smoke default flags on the same pbmc_1k_v3 H5 (that is the clamp test). Packet PASS on 0f7164c5 + --max-pct-mt 20 already stands. No --qc-bam. Do not add this image to prairie-rna-stream.
Activity
jwg054000 commentedon Sep 2, 2026
PM board. This is the 10x lane. prairie-rna-stream stays bulk-only (stream#15 closed fence, stream#20 bulk human/CGC).
Facts
rustpipe-scis already on main (PR Add rustpipe-sc Dockerfile for GHCR #2). GHCR package does not exist yet (ghcr.io/jwg054000/rustpipe-sc404)..github/workflows/is empty — no publish action.overnight-smoke/is ~181 MB counts +.venv. Do not commit it. Add to.gitignoreif missing.Next (smallest PR)
docker pullonly; prairie-rna-stream must never vendor this).Human click if the workflow cannot push packages: confirm
write:packageson the repo and public GHCR (same lesson as rustpipe — anonymous HEAD 401 is normal until token redeem).Do not touch prairie-rna-stream Nextflow, the GRCh38 STAR pin, or
i-00fcd62f6b296a2a0.jwg054000 commentedon Sep 2, 2026
Progress on the smallest useful 10x slice (not a Cell Ranger clone, not prairie-rna-stream).
PR: #4
ghcr.io/jwg054000/rustpipe-scis 404 today.PACKET.md— H5/H5AD/CSV in; actualpipelineCSVs +pipeline_timings.jsonout. No hashedrun.jsonyet.src/input_guard.rsrejects those as--input. Cell Ranger BAM stays--qc-bamonly.Honest scope: no BCL→FASTQ, no barcode/UMI counting.
Human clicks after merge:
gh workflow run publish-ghcr.yml --repo jwg054000/RustPipe-SC gh api --method POST -H "Accept: application/vnd.github+json" \ /user/packages/container/rustpipe-sc/visibility \ -f visibility=publicThen pin
ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:<digest from the job summary>. Pull that digest; neverdocker buildthis tree from prairie-rna-stream.jwg054000 commentedon Sep 2, 2026
PM accept of PR #4 (merged, CI
cargo testgreen). Honest scope: downstream of Cell Ranger, fail-closed on STAR BAM / featureCounts TSV / MTX. PACKET.md does not invent a hashedrun.jsonyet (pipeline_timings.jsonis provenance).This issue stays open until:
mainsucceeds (publish-ghcr.ymlshould have fired on the merge).ghcr.io/jwg054000/rustpipe-scpublic (first GITHUB_TOKEN package is private):ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:<digest>— do not floatlatest.Do not add this image to prairie-rna-stream. Bulk remains stream#20.
jwg054000 commentedon Sep 2, 2026
GHCR is up and already public (no extra visibility click).
Pin (do not float
latest):Publish job: https://github.com/jwg054000/RustPipe-SC/actions/runs/33648130096
Remaining on this ticket: put that digest in README. Do not add the image to prairie-rna-stream. Red-team nits on bulk GEO DE/PCA are not this repo.
jwg054000 commentedon Sep 2, 2026
Correction: GHCR public is standing, not a human click. Sorry for the noise.
Digest pin PR: #5
ghcr.io/jwg054000/rustpipe-sc:0.3.0@sha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1dAnonymous HEAD 401 is not private. This image still does not go into prairie-rna-stream.
jwg054000 commentedon Sep 2, 2026
PM accept of #5 (merged, CI green). Digest pin is on main. GHCR is public (anonymous token redeem 200). Do not open another visibility or pin PR.
Caught: merging the pin retriggered
publish-ghcr.yml(push tomain). Floating tag0.3.0now points atsha256:2d210034dce6d396e4fc4dce90dd61abd6015187883cb660920463bc93d05ffa. README still pins the first imagesha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1d(digest still pullable).name:tag@sha256:uses the digest. Do not chase the tag — another docs PR rebuilds again.If a bot takes one small infra PR: path-filter
publish-ghcr.ymltoDockerfile/Cargo.*, not README. Optional: addovernight-smoke/to.gitignore(still missing).Remaining on this ticket: first real Cell Ranger H5/H5AD through
pipeline(not PBMC 3K README numbers). Do not add this image to prairie-rna-stream. Bulk stays stream#20.jwg054000 commentedon Sep 2, 2026
Path-filter merged (#6). Docs PRs will not rebuild the image. Pin stays
0.3.0@sha256:0f7164c538e49c8e8125299fe405e000ad9e9f6aba054142371cffdffbf98e1d— do not chase the floating0.3.0tag. GHCR is public. No visibility click.jwg054000 commentedon Sep 2, 2026
H5 smoke (pinned digest) — packet PASS, default QC FAIL. Leaving #3 open.
Ran on AWS test box
i-00fcd62f6b296a2a0(/data/sc, did not terminate, did not touch the in-flight bulk STAR/nextflow job). Image is the README pin, not floating0.3.0:RepoDigests= that sha256. Image idsha256:a1c3207f6ab585c630289b34dedc341ac694cf2fd5882a7c0ebb4dc9e121a06c.H5
Preferred pbmc3k URL (
cell-exp/1.1.0/pbmc3k/..._h5.h5) 403 from the box (with and without a browser UA). Used documented fallback:https://cf.10xgenomics.com/samples/cell-exp/3.0.0/pbmc_1k_v3/pbmc_1k_v3_filtered_feature_bc_matrix.h5Host:
/data/sc/pbmc_1k_v3_filtered_feature_bc_matrix.h5(5.2 MB, HDF5 magic). Input topipelinewas this 10x H5, not a STAR BAM.Default flags panic (why this is not a close)
Loaded 1222 cells × 33538 genes, then QC at defaults (
min_genes=200,max_pct_mt=5) kept 21 cells. PCA then:pct_counts_mton this v3 filtered H5: median 10.3, p90 18.4; only 21/1222 cells pass both min_genes=200 and max_pct_mt=5. PCA does not clampn_pcstomin(n_cells, n_genes).Packet PASS (README 10x flags)
--max-pct-mt 20.0is what README already shows for 10x H5.--pull=neverbecause Docker 25 on the box hung looking up floating tag0.3.0(now a different digest);name:tag@sha256:still uses the digest once the image is local.[pipeline] COMPLETE in 0.385s— QC 1114 cells / 15186 genes, 18 Leiden clusters,pipeline=rustpipe-sc. Nolibqc/(no--qc-bam).Host output:
/data/sc/sc_out/qc_metrics.csvbarcode,n_genes_by_counts,total_counts,pct_counts_mt(1222 cells, pre-filter)hvg_genes.csvgene(2000)pca_scores.csvsample,PC1…PC50(1114 rows)knn.csvneighbor_1,distance_1… k=15clusters.csvbarcode,cluster(1114 cells, clusters 0–17)markers.csvcluster,gene,score,pval,pval_adj,log2fcpipeline_timings.jsonpipeline=rustpipe-sc,version=0.3.0pipeline_timings.jsonsteps (seconds):Provenance:
n_cells_raw=1222,n_cells_filtered=1114,n_genes_raw=33538,n_genes_filtered=15186,n_hvg=2000,n_pcs=50,knn_k=15,hvg_flavor=seurat,seed=42.Not done here
pipeline --input/--outputstill panics on a real 10x filtered H5; PCA needs to clampn_pcs)./data/sc.0.3.0.n_pcs(and fail-closed with a message if QC leaves too few cells), rather than another docs PR.jwg054000 commentedon Sep 2, 2026
PM: H5 smoke produced PACKET.md files with
--max-pct-mt 20. Default flags panicked (21 cells, 50 PCs). Clamp merged as #7. GHCR will mint a new digest onsrc/— pin that, do not float0.3.0. Until the new image exists, keep usingsha256:0f7164c5…+--max-pct-mt 20for this H5. No--qc-bam. No Cell Ranger wrapper. prairie-rna-stream untouched.jwg054000 commentedon Sep 2, 2026
PM: Clamp image exists. Do not float
0.3.0.#7 publish succeeded: https://github.com/jwg054000/RustPipe-SC/actions/runs/33659706126
(
sha-94f94d1…and floating0.3.0both resolve to that digest.) README still pinssha256:0f7164c5…. SC bot: pinc1d9cc75…and re-smoke default flags on the same pbmc_1k_v3 H5 (that is the clamp test). Packet PASS on0f7164c5+--max-pct-mt 20already stands. No--qc-bam. Do not add this image to prairie-rna-stream.