Repository navigation
docs(hardware): measure the report step and the clinical pipe before the Rust port - #352
Merged
Merged
Conversation
…the Rust port The runtime table had no time for the two reports (run-all.sh writes them outside the trace) and no breakdown of step 23. Both were measured on the three end-test genomes and are written down with the commands that took them, so the Rust port has a baseline to beat: - the HTML and text reports take 11-15 s each at about 0.4 GB, almost all of it collecting summary.json (each script collects it again); rendering is under half a second; - in step 23's main pipe csq_rank.awk is the saturated stage (about 135 of 157-171 s); bcftools decodes the same stream in 28 s, and --threads 2 changes nothing. Bead: pgp-bs9u.3
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 33 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Bead: pgp-bs9u.3
Guard's image check refuses a literal image tag in docs. Bead: pgp-bs9u.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Before we port OwnGenome's own code to Rust we need numbers to beat. Two were missing from the hardware page. The runtime table said "not measured" for the HTML report, because
run-all.shwrites both reports outside Nextflow's trace. And step 23 had one total time, with no way to tell whetherbin/csq_rank.awk, thebcftoolsdecode or the compression is the slow part. That decides whether porting the awk to Rust makes the step faster at all.Fix
Two new sections in
docs/hardware-requirements.md, each with its numbers and the exact commands that produced them:summary.json, which each of the two scripts does separately (82% of it is reading the small-variant VCF in Python). Rendering from a finishedsummary.jsontakes under half a second. The two reports together take about 25 s, not the 1-3 minutes the page estimated.pvon each edge of the two pipes and timed each stage alone.csq_rank.awkis the saturated stage: about 135 s of the main pipe's 157-171 s.bcftoolsdecodes the same 20 GiB stream in 28 s and compresses the result in 2.5 s. Adding--threads 2to bothbcftools viewcalls makes no difference (154-173 s, same records), so that one-flag change is not worth shipping. A faster ranker could bring the pipe down toward the decode time, about 25-30 s.Rows 23 and 24 of the runtime table now link to these sections. Row 24 keeps its estimate as the end-test figure, because
assets/steps.tsvfeeds that estimate to the ETA and this PR does not change it.No code, option, config key or output changes. The docs contain only aggregate numbers and commands.
Decisions:
docs/owngenome-baseline-2026-11.mdpage. A new page also needs a nav entry inzensical.toml, or the site check fails. The plan's lane note puts these numbers indocs/hardware-requirements.mdanyway, so they go there and nothing else changes.CHANGELOG.mdin the repo, so this PR adds none.E2E_TRACE=true). It takes one serial runner leg of about 4-6 hours and records only per-case times and which files each case reads, which the sharded runs already give us. It measures neither the report step nor the clinical pipe.How it was checked
python -m zensical build --clean --strictand.github/scripts/site-check.py sitepass locally: 95 pages, 0 errors. Red control: with the new#the-report-steplink changed to a missing anchor, the strict build fails with "anchor does not exist".scripts/ci/check-personal-data.shandscripts/ci/check-name.shpass locally.BCFTOOLS_IMAGE, andscripts/ci/check-images.shpasses.Bead: pgp-bs9u.3