Fix #700: invalidate provider cache when its config changes#777
Open
phverg wants to merge 1 commit into
Open
Conversation
``Provider.status()`` only checked whether the cache directory existed on disk. Once a core had been fetched, FuseSoC happily reused the cached files for every subsequent run, even if the user had since changed the core's ``provider`` block — most notably the ``version`` field. So flipping ``version: v1.0.4-rc0`` to ``version: v1.0.4-rc1`` silently kept building against the rc0 sources, with the only workaround being to bump the core's own VLNV version. Persist a ``.fusesoc-provider-config.json`` marker inside the cache directory after each successful fetch. ``status()`` now reads it and flags the cache as ``outofdate`` whenever the current provider config no longer matches what was cached, which then triggers the existing clean-and-refetch path. Caches written by older FuseSoC versions don't have a marker; treat those as valid for backward compatibility and let the next fetch write a fresh marker so any future drift will be caught. Fixes olofk#700. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
|
This makes sense, but I need to think if there are some other implications of this. About to release a new FuseSoC version, so this fix will likely land after that. |
5 tasks
phverg
added a commit
to phverg/fusesoc
that referenced
this pull request
May 17, 2026
``Ttptttg._sha256_input_yaml_hexdigest`` hashed only
``{parameters, vlnv, gapi}`` from ``generator_input``. Editing
the generator -- whether by pointing ``command:``/``interpreter:``
at a different file or by editing the script's bytes in place --
left the hash identical, so subsequent runs silently served the
previously-cached output even though the program that produced
it had changed.
Fold the generator definition into the hash:
* ``command`` -- declared script path
* ``interpreter`` -- declared interpreter name
* SHA-256 of the resolved script bytes -- closes the in-place
edit gap (same path, same declarations, new bytes). Same
pattern branch olofk#777 follow-ups use for provider patch bytes.
Deliberately NOT hashed:
* The resolved interpreter path from ``shutil.which(interpreter)``
-- depends on the system PATH at run time and is not part of
the generator definition the user signs in their .core file.
* The generator's ``root`` path -- it's an absolute path that
differs between machines/checkouts, which would make hashes
non-portable and cache directory names environment-specific.
Two generators with the same relative ``command`` rooted under
different directories are still distinguished by the script
bytes hash: same bytes means functionally equivalent output,
so sharing the cache is the desired behaviour.
This necessarily invalidates existing on-disk caches for
cacheable generators on first run after upgrade (the directory
name embeds the hash). That's intentional: users get one extra
regeneration and consistent behaviour afterwards. The hardcoded
reference hashes in ``test_generators`` are updated accordingly.
Fixes olofk#751.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #700.
Provider.status()only checks whether the cache directory exists on disk. Once a core has been fetched, FuseSoC happily reuses the cached files for every subsequent run, even if the user has since changed the core'sprovider:block — most notably theversionfield. So flippingversion: v1.0.4-rc0→version: v1.0.4-rc1silently keeps building against the rc0 sources, with the only workaround being to bump the core's own VLNV version (which then cascades through dependents).Solution
After a successful fetch, write a
.fusesoc-provider-config.jsonmarker inside the cache directory containing a normalised JSON copy of the provider config.status()reads it back on subsequent runs and flags the cache asoutofdatewhenever the current config no longer matches, which then triggers the existing clean-and-refetch path.Caches written by older FuseSoC don't have the marker; treat those as valid for backward compatibility (no surprise refetch on upgrade) and let the next fetch write a fresh marker so any future drift is caught.
Test plan
test_provider_cache_invalidates_on_config_change— confirms a changedversionflips status fromdownloadedtooutofdate.test_provider_legacy_cache_without_marker_is_valid— confirms a pre-marker cache is still treated asdownloaded.test_git_provider,test_url_provider,test_uncachable, etc.) still pass.pre-commit run -aclean.