Repository navigation
Key GCP folds run ids on output-changing options - #4
Merged
Merged
Conversation
hound_snp_folds, hound_ism_snp_folds and hound_ism_bed_folds derived their deterministic GCS output dir (and so per-shard resume) from the input and model hashes only. Rerunning with different options (precision, stats, targets, --rc, job size, ...) silently reused the earlier run's completed shards, e.g. a -m bfloat16 rerun merged fp32 shards of another job size. The run id now also hashes a representative job command (placeholder model/output/fold paths), so any option change starts a fresh prefix while identical reruns still resume. Adds stage_cache.hash_text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hound_eval_folds and hound_eval_genes_folds keyed their deterministic GCS output dir on models, params, (gtf,) and data only, so rerunning with different options (--rc, --shifts, --step, ...) resumed the earlier run's finished acc.txt / gene_metrics.tsv. Their output-changing flags move into eval_options/spec_options/ evalg_options, used both to build each job command and in the run id. Per-job paths and splits, and spec's memory/IO knobs (--ncpus, --band, --seq_chunk, --ram, --scratch_dir), stay out of the key so changing them still resumes. Spec options are keyed even without --spec, so adding --spec to a finished eval run resumes it. Gene eval also keys --valid, which changes the split written to the same evalg/ dir. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On GCP, the
_foldswrappers pick a deterministic output dir and resume any job whose output is already complete. That dir was keyed only on inputs (VCF/BED, models, params, data), so rerunning with different options silently reused the earlier run's results. In practice, ahound_snp -m bfloat16 -j 6000rerun merged 12 fp32 shards from an earlier-j 1024run. With the same-j, it would have silently returned fp32 scores.The run id now also hashes the options that change the output:
hound_snp_folds,hound_ism_snp_folds,hound_ism_bed_folds: hash a representative job command (placeholder model/output/fold paths). Every flag changes output, and job size must match for per-shard resume.hound_eval_folds: option flags move intoeval_options/spec_options, used both to build the commands and in the key. Spec's memory/IO knobs (--ncpus,--band,--seq_chunk,--ram,--scratch_dir) stay unkeyed, so changing them still resumes. Spec options are keyed even without--spec, so adding--specto a finished eval resumes it.hound_eval_genes_folds: same, viaevalg_options, plus--valid, which changes the split written to the sameevalg/dir.Adds
stage_cache.hash_text.Note: existing runs get new output dirs, so an in-progress GCP run won't resume across this change.
Test plan
-m,-j,--rc,--shifts,--valid), and the same dir for memory/IO knobs and for adding--spec. All fail on the old code.test_snp_folds,test_stage_cache,test_ism_snp,test_ism_bed,test_gcprunner,test_eval,test_hound_eval_genes,test_eval_*_folds_gcp🤖 Generated with Claude Code