Repository navigation
Add @-relative includes, @module/path includes, $ENV expansion, and path:key fragment extraction - #54
tiagotaveiragomes wants to merge 4 commits into
Conversation
A leading '@' in an include urlpath now resolves relative to the directory of the file doing the including, instead of base_dir / the current working directory (falls back to cwd at the top level, where there is no including file yet). Env vars embedded in urlpaths (e.g. $HOME/foo.yaml) are expanded via os.path.expandvars before any other resolution.
'@modulename/relative/path.yaml' resolves relative to the imported module's package directory (importlib.import_module(modulename).__path__[0]). If modulename isn't importable, falls back to plain @-relative resolution, so '@include.d/1.yaml'-style paths (module-name-shaped but not actually a module) keep working.
'some/file.yaml:key.subkey' loads the file, then descends into the loaded mapping via key.subkey (split on '.'), returning only that nested value. Only applies to plain local, non-wildcard paths; real URLs (which always contain '://') and glob patterns are left alone.
Address reviewer findings against the sigil upstream feature branch: 1. SECURITY: @modulename/path no longer implicitly imports arbitrary installed packages as a side effect of yaml.load. Gated behind a new Constructor(allow_module_include=False) opt-in flag (default False); when disabled, @modulename/path falls through to plain @-relative resolution instead of raising. 2. Non-package modules (e.g. os.path, no __path__) now fall back to @-relative resolution instead of raising a raw AttributeError. 3. Namespace packages with multiple __path__ entries are now resolved by trying every entry (first that exists wins), not just entry [0]. 4. path:key.subkey fragment splitting no longer misparses a Windows drive letter (C:\... / C:/...) as a fragment separator. 5. _including_file_dir is now tracked correctly for files reached via a wildcard/glob include (set per matched file); for URL-scheme includes it's explicitly reset rather than left stale, so a nested @-relative include can't silently resolve against the wrong directory. Also documents all 4 new include features (@-relative, @module/path + its opt-in flag, $ENV expansion, path:key.subkey fragments) in the Constructor docstring, and adds a CHANGELOG entry. Known limitations left as-is (out of scope for this PR): fragment extraction only supports dotted mapping keys, not list indices; env-var expansion is unconditional (no opt-out). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Thank you for putting this together. There are several genuinely useful ideas here, and the amount of attention paid to tests and edge cases is appreciated. In particular, making module inclusion opt-in, restoring nested state in After reviewing the proposal, however, I do not think we should merge it in its current form. My main concerns are not isolated implementation details; they are about the semantic boundary between YAML, PyYAML, and the filesystem abstraction provided by this project. 1. The fundamental source-context limitationYAML itself has no concept of:
At the PyYAML level, a constructor is called with a loader and a node. The input may have originated from a local file, but it may equally be a string, current_document_uri = ...Even if a particular file object has a This means that the source location of the root YAML document is generally unavailable to The situation is somewhat different for documents opened by The proposed If
Even then, this distinction between root and nested documents would need prominent documentation. 2.
|
|
Thank you for coming back to me! With great feedback 😀 |
Summary
Four related additions to include-path resolution, each as its own commit:
@-relative includes: a urlpath starting with@resolves relative to the directory of the file doing the including, rather than the process cwd. Useful for includes nested several directories deep where cwd-relative paths are fragile.$ENVexpansion: urlpaths are passed throughos.path.expandvars(...), so!inc $CONFIG_DIR/foo.yamlpicks up environment variables.@modulename/pathincludes: resolve a urlpath relative to an installed Python module's__path__viaimportlib, for referencing config shipped inside a package. Handles non-package modules (falls back cleanly instead of raising) and namespace packages with multiple__path__entries (tries each). Opt-in: gated behindConstructor(..., allow_module_include=True), defaultFalse— importing a module runs its top-level code, so this changes the trust model ofyaml.loadand should not be on by default.path:key.subkeyfragment extraction:!inc some/file.yaml:key.subkeyloads the file, then descends into the parsed result via dotted-key lookup, so a single include can pull out a nested value rather than the whole document. Carefully scoped to avoid colliding with real URL schemes (://), glob patterns, and Windows drive letters (C:\...,D:/...).Testing
Full existing suite passes with no regressions (450 passed, up from 359 baseline on
main), plus new tests per feature covering the edge cases above (non-package modules, multi-path namespace packages, Windows-drive-letter vs. fragment-colon disambiguation,@-relative resolution inside glob/wildcard and URL-scheme includes with no state leakage between sibling includes).Notes
CHANGELOG.mdand theConstructorclass docstring are updated to document all four features, including the security rationale forallow_module_includedefaulting toFalse.$ENVexpansion is currently unconditional (no opt-out).Happy to split this into separate PRs per feature if preferred, or adjust the
allow_module_includedefault/naming — open to your take on the API shape.