Cache initialize_diffcheck cost - #28
Open
ArjunS07 wants to merge 13 commits into
Open
Conversation
Resource-keyed cache of LSV baseline timings so harbor_healthcheck can skip the base-commit timing pass on repeat trials. Populated by the oracle trial, read by every trial. Private (grafana_ro SELECT only), no FK (advisory).
…hit) Stage the pre-baked deps DB, fetch resource-keyed baselines from lsv_baseline_cache on datasmith's Supabase, and pass force=False only on a full hit so initialize_diffcheck skips the survey+timing passes. Any miss or missing creds falls through to force=True (unchanged behaviour). Records the key in lsv_resource_attrs.json and hit/miss in lsv_cache_state.json for the writeback, and exports FORMULACODE_BASELINE_FROM_CACHE for invariant #20. Preserves the baseline_sha breadcrumb and --repeat/--warmup-time.
Oracle-only writeback that upserts freshly measured baselines into lsv_baseline_cache when lsv_init recorded a miss. Reads the resource key from lsv_resource_attrs.json (single source of truth with lsv_init) and the timings from the deps DB baseline table. Stdlib only; best-effort, never fails the trial. Not yet baked/invoked -- wiring follows.
Add a render_env dict that renders as shell-quoted export lines in setup.sh and test.sh, so the LSV cache creds and resource key reach lsv_init (setup.sh, which Harbor's [verifier.env] misses on Daytona) and the writeback (test.sh). Register a shell_quote Jinja filter, thread render_env through generate_task, bake lsv_cache_writeback.py into the image, and invoke it oracle-only after the reward parser. render_env defaults to empty -> byte-identical output when the runner has not enabled the cache.
Inject datasmith Supabase creds (DATASMITH_-prefixed) and the resource key (env, container_name, image_digest, machine_class/docker_host_id, cpu/mem pins) into each task's render_env so lsv_init can look baselines up and the writeback can upsert them. Unify the trial cpu/memory pins with the cache key via _trial_pins so the key describes the hardware the trial actually got (memory default stays 32 GB on both environments). Gated on DATASMITH_LSV_CACHE_ENABLED and the presence of creds; off => byte-identical to prior behaviour.
Author
|
This requires running the following migrations on the Datasmith postgres @atharvas :
Merging without these migrations will lead to no-ops and print a warning |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During
setup.sh, LSV invokesinitialize_diffcheck. This:We now introduce 2 new database tables:
lsv_baseline_cache: keeps the baseline timings. Its key has 11 parts. The key includes the CPU model from inside the container. A timing is used again only on the same hardware.For every task, when the cache is enabled, the runner (which launches containers) does the following
_fetch_deps_db(owner, repo, issue_number), which queries lsv_deps_cache.environment/cache/lightspeed_deps.db)The container then spawns. It:
If any one result is absent, the container measures up to sufficiency again.
The only thing that can write the cache is an oracle run.