FYI: related project fscacher/question

[yarikoptic]

Just ran into this library which is of interest, yet to try/explore more. But it resonated with me as potentially (?) to replace our home-brewed joblib based https://github.com/con/fscacher where the goal is to provide easy means to cache results of compute on files (think e.g. extraction of metadata or even checksums). Does exca provide such a facility (caching based on the state of the file)?

[kingjr]

Thanks for the pointer. I didn't quite get what functionality you're looking for, would you have an example use case?

[yarikoptic]

taking on the example in your README could be smth like

  class MyTask(pydantic.BaseModel):
      path: Path = xk.Path
      infra: xk.TaskInfra = xk.TaskInfra()

      @infra.apply
      def process_file(self) -> str:
          return hashlib.sha256(self.path.read_bytes()).hexdigest()

so the point here is that if path content changes (according to change in some fingerprint e.g. size, mtime, inode) - function is recomputed but if not -- result is cached/reused

[kingjr]

I see. I think this should can covered with pydantic api:

from pathlib import Path
import hashlib
import pydantic

class MyTask(pydantic.BaseModel):
    path_uid: str
    infra: xk.TaskInfra = xk.TaskInfra()

    @pydantic.field_validator("path_uid", mode="before")
    @classmethod
    def hash_path(cls, v):
        return hashlib.sha256(Path(v).read_bytes()).hexdigest() # potentially concatenate it to `v` 

    @infra.apply
    def run(self) -> tp.Any:
        # your main compute here
        return 'foo'

The downside is that it'd check at the initialization, whereas exca usually check for cache at execution.

[yarikoptic]

what in above would take care about invalidating the cache when file fingerprint (e.g. mtime or size) changes?

[kingjr]

At init, my snippet gets the hexdigest of the file and keep it as param in the class.

The exca infra uses the non-default param to compute the unique id, against which the infra is checked. If that unique id isn't already cached, then the method will be executed.