Skip to content

Add @-relative includes, @module/path includes, $ENV expansion, and path:key fragment extraction - #54

Closed
tiagotaveiragomes wants to merge 4 commits into
tanbro:mainfrom
sigil-enterprises:feat/sigil-upstream-features
Closed

tiagotaveiragomes wants to merge 4 commits into
tanbro:mainfrom
sigil-enterprises:feat/sigil-upstream-features

Conversation

@tiagotaveiragomes

Copy link
Copy Markdown

Summary

Four related additions to include-path resolution, each as its own commit:

  • @-relative includes: a urlpath starting with @ resolves relative to the directory of the file doing the including, rather than the process cwd. Useful for includes nested several directories deep where cwd-relative paths are fragile.
  • $ENV expansion: urlpaths are passed through os.path.expandvars(...), so !inc $CONFIG_DIR/foo.yaml picks up environment variables.
  • @modulename/path includes: resolve a urlpath relative to an installed Python module's __path__ via importlib, for referencing config shipped inside a package. Handles non-package modules (falls back cleanly instead of raising) and namespace packages with multiple __path__ entries (tries each). Opt-in: gated behind Constructor(..., allow_module_include=True), default False — importing a module runs its top-level code, so this changes the trust model of yaml.load and should not be on by default.
  • path:key.subkey fragment extraction: !inc some/file.yaml:key.subkey loads the file, then descends into the parsed result via dotted-key lookup, so a single include can pull out a nested value rather than the whole document. Carefully scoped to avoid colliding with real URL schemes (://), glob patterns, and Windows drive letters (C:\..., D:/...).

Testing

Full existing suite passes with no regressions (450 passed, up from 359 baseline on main), plus new tests per feature covering the edge cases above (non-package modules, multi-path namespace packages, Windows-drive-letter vs. fragment-colon disambiguation, @-relative resolution inside glob/wildcard and URL-scheme includes with no state leakage between sibling includes).

Notes

  • CHANGELOG.md and the Constructor class docstring are updated to document all four features, including the security rationale for allow_module_include defaulting to False.
  • Known limitations left out of scope to keep this focused: fragment extraction supports dotted mapping keys only (no list indices); $ENV expansion is currently unconditional (no opt-out).

Happy to split this into separate PRs per feature if preferred, or adjust the allow_module_include default/naming — open to your take on the API shape.

tiagotaveiragomes and others added 4 commits August 31, 2026 00:34
A leading '@' in an include urlpath now resolves relative to the
directory of the file doing the including, instead of base_dir /
the current working directory (falls back to cwd at the top level,
where there is no including file yet). Env vars embedded in urlpaths
(e.g. $HOME/foo.yaml) are expanded via os.path.expandvars before
any other resolution.
'@modulename/relative/path.yaml' resolves relative to the imported
module's package directory (importlib.import_module(modulename).__path__[0]).
If modulename isn't importable, falls back to plain @-relative
resolution, so '@include.d/1.yaml'-style paths (module-name-shaped
but not actually a module) keep working.
'some/file.yaml:key.subkey' loads the file, then descends into the
loaded mapping via key.subkey (split on '.'), returning only that
nested value. Only applies to plain local, non-wildcard paths; real
URLs (which always contain '://') and glob patterns are left alone.
Address reviewer findings against the sigil upstream feature branch:

1. SECURITY: @modulename/path no longer implicitly imports arbitrary
   installed packages as a side effect of yaml.load. Gated behind a new
   Constructor(allow_module_include=False) opt-in flag (default False);
   when disabled, @modulename/path falls through to plain @-relative
   resolution instead of raising.
2. Non-package modules (e.g. os.path, no __path__) now fall back to
   @-relative resolution instead of raising a raw AttributeError.
3. Namespace packages with multiple __path__ entries are now resolved by
   trying every entry (first that exists wins), not just entry [0].
4. path:key.subkey fragment splitting no longer misparses a Windows
   drive letter (C:\... / C:/...) as a fragment separator.
5. _including_file_dir is now tracked correctly for files reached via a
   wildcard/glob include (set per matched file); for URL-scheme includes
   it's explicitly reset rather than left stale, so a nested @-relative
   include can't silently resolve against the wrong directory.

Also documents all 4 new include features (@-relative, @module/path +
its opt-in flag, $ENV expansion, path:key.subkey fragments) in the
Constructor docstring, and adds a CHANGELOG entry.

Known limitations left as-is (out of scope for this PR): fragment
extraction only supports dotted mapping keys, not list indices; env-var
expansion is unconditional (no opt-out).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@tanbro

tanbro commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Thank you for putting this together. There are several genuinely useful ideas here, and the amount of attention paid to tests and edge cases is appreciated. In particular, making module inclusion opt-in, restoring nested state in finally blocks, testing namespace packages, and considering Windows drive-letter ambiguity all show careful thought.

After reviewing the proposal, however, I do not think we should merge it in its current form. My main concerns are not isolated implementation details; they are about the semantic boundary between YAML, PyYAML, and the filesystem abstraction provided by this project.

1. The fundamental source-context limitation

YAML itself has no concept of:

  • including another document;
  • the URI from which a document was loaded;
  • a "current file" or "current directory";
  • filesystem-relative resolution.

!inc is entirely an application-defined extension.

At the PyYAML level, a constructor is called with a loader and a node. The input may have originated from a local file, but it may equally be a string, StringIO, stdin, an HTTP response, or another arbitrary stream. There is no stable PyYAML contract that tells the constructor:

current_document_uri = ...

Even if a particular file object has a .name attribute, that is not a reliable resource identifier and does not generalize to fsspec filesystems or arbitrary streams.

This means that the source location of the root YAML document is generally unavailable to Constructor. The existing base_dir option is important precisely because it makes that missing context explicit and deterministic: the caller supplies the root against which relative include paths should be resolved.

The situation is somewhat different for documents opened by pyyaml-include itself. Once this library has opened an included resource, it knows the resource path while recursively parsing it, so it can maintain a stack of nested resource contexts. However, that still does not solve the root-document problem, and it requires tracking more than a local directory string.

The proposed @ behavior currently handles the missing root context by falling back to the process working directory. I do not think that is a good semantic fallback. It makes a syntax intended to provide stable document-relative resolution depend once again on how the application was launched. It also bypasses an explicitly configured base_dir.

If @ is retained, the root behavior would need to be defined explicitly, for example:

  • use the current resource when one is known;
  • otherwise use base_dir;
  • if neither is available, raise a clear error.

Even then, this distinction between root and nested documents would need prominent documentation.

2. @ currently has two ambiguous meanings

The PR uses the same prefix for two different concepts:

!inc @directory/file.yaml
!inc @package_name/data/file.yaml

The first means "relative to the including resource," while the second may mean "relative to an importable Python package."

The interpretation of this input:

!inc @requests/defaults.yaml

can therefore change depending on:

  • whether allow_module_include is enabled;
  • whether a package named requests happens to be installed;
  • whether that object is a package or a plain module;
  • whether the requested resource exists in the package;
  • potentially which namespace-package path contains the resource.

That makes the meaning of the YAML dependent on the Python environment in a way that is difficult to see from the document itself. Installing an unrelated package could change an existing path from a filesystem-relative include into a package-relative include.

The fallback behavior makes this more ambiguous rather than less ambiguous. If package resolution fails, the same string is reinterpreted as a normal @-relative path. A typo in a package name and an intentional relative directory therefore have indistinguishable behavior.

I think @, if introduced at all, should represent only one concept: resolution relative to the current resource. Python package resources need an explicit and unambiguous syntax, such as a structured form:

value: !inc
  package: my_package
  path: data/default.yaml

A dedicated package:// form might also be considered, although that raises a separate question about whether it should be implemented as an fsspec protocol. In either case, package resolution should not compete with ordinary relative paths for the meaning of @.

3. The proposed @ implementation does not preserve fsspec semantics

This project advertises local and remote resource support through fsspec. Therefore, "relative to the including file" cannot reasonably mean only "relative to a local directory represented by pathlib.Path."

Consider a remote document:

https://example.com/config/main.yaml

containing:

database: !inc @database.yaml

The natural interpretation is:

https://example.com/config/database.yaml

The current implementation explicitly clears _including_file_dir while loading URL-scheme resources. The nested @database.yaml therefore falls back to the local process working directory. A remote document can unexpectedly cause a local file to be opened, and the same YAML expression has fundamentally different meanings in local and remote documents.

This is more than a missing optional capability: it contradicts the general description that @ resolves relative to the file performing the include.

The same concern applies when a caller supplies an fsspec filesystem directly:

Constructor(
    fs=s3_fs,
    base_dir="bucket/config",
)

In this mode, paths may not contain an explicit s3:// scheme; they are names in the configured filesystem's namespace. Using pathlib.Path, Path.cwd(), and host-operating-system path rules to resolve them is not a sound abstraction. It may happen to work for simple POSIX-shaped object names, but it is not guaranteed for S3, HTTP, memory, archive, or custom filesystems.

A correct nested-resource context would need to preserve at least:

  • the filesystem instance;
  • the protocol, if applicable;
  • the full resource path;
  • the resource's parent path in that filesystem;
  • any filesystem configuration or credentials already carried by the instance.

A nested relative include should normally remain on the same filesystem and be resolved according to that filesystem's path model. A directory string alone is insufficient.

The context should also be stack- or context-based rather than shared mutable instance state. _including_file_dir is restored correctly for single-threaded recursion, but the same Constructor can be registered globally and used by concurrent loads. Concurrent parses could overwrite one another's current directory. A ContextVar or an explicit per-load context would be safer.

4. There is a concrete Windows failure

After resolving an @ or package-relative path, the implementation calls urlsplit() on the resulting path.

On Windows:

urlsplit("C:/config/file.yaml")

interprets C as the URI scheme:

scheme = "c"

The code will then take the URL/fsspec branch instead of the configured local-filesystem branch, potentially attempting to open an unknown c protocol.

The tests verify that the fragment regular expression does not treat a Windows drive colon as a fragment separator, but they do not exercise the complete @ or module-relative loading path on Windows. An end-to-end Windows test is necessary here.

More generally, this demonstrates the danger of mixing three different path models without a clear boundary:

  • local paths through pathlib;
  • URIs through urllib.parse;
  • filesystem paths and protocols through fsspec.

Scheme detection and filesystem resolution should use fsspec-aware utilities where possible, while still taking care not to classify Windows drive letters as protocols.

5. Python package resources should use the package-resource abstraction

Resolving package data through:

importlib.import_module(name).__path__

and then treating it as an ordinary local directory is not the usual Python package-resource abstraction.

importlib.resources exists to support resources from:

  • ordinary filesystem packages;
  • namespace packages;
  • zip-based imports;
  • importers whose resources are not represented by normal local paths.

Using module.__path__ and Path.exists() unnecessarily restricts the feature to packages with real filesystem directories.

There is also an error-handling issue: catching ModuleNotFoundError around the whole import_module() call can suppress an error raised by a missing dependency inside a package that does exist. Such an import failure would then be reinterpreted as a relative-path include, hiding the real package error.

The opt-in security gate is the right instinct, but package-resource inclusion should be a separate feature with separate syntax, based on importlib.resources, and preferably proposed independently.

6. Environment expansion changes every existing path implicitly

Environment-variable expansion is useful, especially in deployment-oriented configuration. However, applying os.path.expandvars() unconditionally to every include path changes the interpretation of existing documents.

Some concerns are:

  • results become dependent on process environment;
  • unresolved variables are silently retained;
  • behavior is platform-dependent;
  • literal dollar-sign paths may change meaning;
  • callers have no way to disable expansion.

Many configuration systems support environment interpolation, but it is usually an explicit feature or a preprocessing layer owned by the application. If this project adds it, I would prefer an opt-in constructor option or explicit YAML form rather than unconditional expansion. This feature can also be reviewed independently of resource-relative resolution.

7. path:key.subkey combines resource location and object selection

Selecting a subtree from an included document is useful, but path:key.subkey introduces another grammar into the path string. It must distinguish selection syntax from:

  • URI schemes;
  • Windows drive letters;
  • valid filenames containing colons;
  • wildcards;
  • future fsspec protocols;
  • potentially URI fragments and query strings.

The increasing number of regular-expression exclusions is a sign that resource location and document selection are separate concerns.

A structured form would be clearer and extensible:

value: !inc
  urlpath: config.yaml
  select: top.mid.leaf

Alternatively, a standard pointer syntax such as JSON Pointer could be considered, but URI-fragment interaction would still need careful design. In either case, I recommend treating subtree selection as a separate proposal.

Suggested way forward

I do not want to reject the underlying use cases. In particular, nested resource-relative includes would be valuable for reusable configuration trees. However, I suggest splitting this PR and first defining a resource-resolution model that works consistently with the existing fs and base_dir API.

Before implementing @, we should answer these questions explicitly:

  1. What does @child.yaml mean in the root document when the root source is unknown?
  2. What does it mean when base_dir is set?
  3. What does it mean inside a local included file?
  4. What does it mean inside file://, HTTP, S3, memory, archive, and custom fsspec resources?
  5. Does it always remain on the filesystem of the containing resource?
  6. How is context isolated across nested, wildcard, and concurrent loads?
  7. What happens when the current resource has no meaningful parent?
  8. Is local-only behavior acceptable for a library whose public abstraction is fsspec?

A focused implementation should then have end-to-end coverage for at least:

  • POSIX local paths;
  • Windows local paths;
  • an explicit file:// URI;
  • an HTTP document containing a nested relative include;
  • MemoryFileSystem or another non-local configured filesystem;
  • glob results where each matched resource has its own parent;
  • exception restoration;
  • concurrent use of a shared Constructor, if that remains supported.

The other three features - package resources, environment expansion, and subtree selection - should be proposed separately with unambiguous APIs.

In summary, the use cases are valid, and parts of the implementation show good care, but the current PR crosses several abstraction boundaries at once. The proposed behavior is currently local-filesystem-oriented, while pyyaml-include exposes a broader fsspec resource model. It also assigns multiple environment-dependent meanings to @.

For those reasons, I am not comfortable merging the PR as it stands. I would be happy to continue with a narrower proposal focused first on a well-defined, filesystem-independent model for nested resource-relative includes.

@tiagotaveiragomes

Copy link
Copy Markdown
Author

Thank you for coming back to me! With great feedback 😀
I am happy to put work on reviewing implementation to go exactly towards your view. My contribution comes from a very narrow scoped use-case so it misses the implications for the broader picture and technologies to be supported. Please let me know and I will put in the effort 😀

@tanbro tanbro closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants