Skip to content

Inconsistency between configuration of location for local and remote data trees #32

@tlestang

Description

@tlestang

When fetching or writing to a remote data tree, the path to the file is assembled by the ArchiveStore._inarchiveformatpath method as

remote["root"] / self.storehead / remote["path"]

the whole think is then passed to e.g. storage.Archive.retrieve. By default the storehead attribute is "vortex". The value of remote["root"] is read form configuration (storage:rootdir). This means that if the remote data tree is located at /a/b/c/vortex, then the storage:rootdir configuration should be set to /a/b/c which is technicaly not the root of the data tree.

This is in contrast with the local data tree location, for which the configuration value must be the actual data tree root directory.

This inconsistency should be ironed out. Quickest way would be to get rid of the storehead attribute. Ideally, the remote data tree location would be set at the level of the Archive object, in the same way the local data tree location is set at the level of the Cache object (see below).


The location of the local data store is set by the tools.storage.Cache's entry attribute. This is set whenever a CacheStore object calls _get_cache by passing self.location as a value for entry. So the cache location is set at the level of the cache. When calling incacheget, whatever AbstractStore object passes the part of the file path below cache location to Storage.retrieve:

    def incacheget(self, remote, local, options):
        rc = self.cache.retrieve(
            remote["path"],
            local,
            # ...
        )

where remote is e.g.

{'path': '/arome/pefrance/H19P/20250205T2100P/mb002/forecast/grid.arome-forecast.eurw1s40+0001:00.grib', 'params': '', 'query': {}, 'fragment': '', 'port': None, 'username': None, 'password': None}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions