Proposal: tool-independent embedded metadata with YAML/TOML START/END markers

[DandyLyons]

Summary

Introduce embedded metadata: a tool-independent, explicitly marked file-level metadata block carried by host-language comments. Use format-specific markers that identify both the serialization format and the block boundaries, with no inner frontmatter fences:

*** YAML START ***
... YAML payload ...
*** YAML END ***

*** TOML START ***
... TOML payload ...
*** TOML END ***

Decouple recognition, host comment wrapping, and insertion placement. A block's identity must no longer depend on being at the beginning of the file.

This is a proposed specification for implementation, not an already established external standard. The syntax deliberately contains no application name: metadata belongs to the file and its author, and other tools should be able to implement this format independently.

Motivation and scope

The current top-of-file model accumulates exceptions: shebangs, Swift tools-version declarations, Python encoding declarations and module documentation, XML declarations, and project license-header requirements. Moving everything to EOF would create another collection of exceptions, including editor modelines, signatures, and trailing data sections.

Use embedded metadata for the feature, metadata block for an occurrence, and comment-embedded metadata when discussing representation. Reserve header/footer for placement and frontmatter for traditional document frontmatter. METADATA is implied by the convention and does not appear in the markers.

This proposal covers non-Markdown host files and their comment representations. Ordinary Markdown frontmatter keeps its existing syntax and recognition behavior. Metadata keys and values remain user-defined, subject to the existing YAML/TOML metadata data model; this envelope adds no required keys, application namespace, or inferred type.

The nontraditional-file feature is unreleased and migration support is not required. Do not add legacy fallback scanning, dual-format precedence, or migration machinery for the previous unmarked non-Markdown representation.

Proposed specification

MUST, MUST NOT, and SHOULD below describe the proposed conformance rules. Additional product choices that are not yet settled are listed separately at the end.

1. Logical structure

After removing the host comment wrapper, a complete block consists of an opening marker, the raw YAML or TOML payload, and its matching closing marker:

Format	Opening marker	Closing marker
YAML	*** YAML START ***	*** YAML END ***
TOML	*** TOML START ***	*** TOML END ***

Each marker MUST occupy its own logical line and begin at logical column one after wrapper removal. Capitalization and internal spacing are exact. Trailing ASCII spaces/tabs are allowed; leading logical indentation is not. Canonical writers emit markers without trailing whitespace. No suffix text or inline annotation is allowed.

The format-specific markers replace both the earlier METADATA markers and inner --- / +++ fences. Do not support the earlier proposal as an alternative representation: no migration or compatibility path is needed.

The complete interior is passed directly to the selected YAML/TOML payload parser after wrapper removal. Markers are excluded. No frontmatter fences are required, inserted, or automatically stripped. Any --- within a YAML payload belongs to YAML syntax, not envelope syntax; +++ has no envelope meaning either. Retain the existing metadata data model and require a single YAML document, rather than introducing multi-document metadata.

2. Whitespace and line endings

Empty and whitespace-only lines MUST be permitted both at the beginning/end of the payload inside the markers and within the payload. For envelope purposes, whitespace-only means zero or more ASCII spaces or tabs after wrapper removal.

Do not remove, trim, or collapse payload whitespace. Indentation matters to YAML, and apparent blank lines can belong to YAML block scalars or TOML multiline strings. Whitespace acceptance does not override the underlying serialization grammar: invalid YAML/TOML must still receive a payload diagnostic.

Keep the existing LF-only policy. CRLF delimiters are not recognized; do not add CRLF support or normalize unsupported line endings during writes. Preserve an optional initial UTF-8 BOM and support a final closing line without a newline.

3. Hash-comment representation

# *** YAML START ***
# title: Git exclusions
# owner: platform
# tags:
#   - git
#   - configuration
# *** YAML END ***

For the hash profile, remove exactly one # prefix from each nonempty content line. A bare # represents an empty logical line. Physically empty or ASCII-whitespace-only lines are also accepted within the envelope. Every other physical line MUST have the required comment prefix; an unprefixed source-code line inside the block is an error.

For the initial profile, comment prefixes begin at column one. Spaces after the single separator space are payload, not incidental indentation. This avoids silently dedenting YAML. Canonical writers use # for content lines and # for empty logical lines; whitespace-bearing payload lines retain their whitespace after # .

TOML uses the same envelope:

# *** TOML START ***
# title = "Git exclusions"
# owner = "platform"
# tags = ["git", "configuration"]
# *** TOML END ***

Use this representation for .py and .pyi instead of Python triple-quoted strings. Python strings are not generic comments: inserting them can replace module documentation or disrupt future-import ordering. Preserve shebangs, encoding declarations, the original module docstring, and future imports.

Existing hash-compatible file mappings, including .gitignore, continue to determine eligible host files. A marker does not make hash comments legal in a language that lacks them.

4. Block-comment representation

The outer host-comment opening and closing tokens MUST occupy dedicated physical lines. Within that wrapper, the same logical marker/payload structure applies, without hash prefixes:

<!--
*** YAML START ***
title: Example
*** YAML END ***
-->

Only whitespace-only lines may occur between the opening host wrapper and opening metadata marker, or between the closing metadata marker and closing host wrapper. Use a dedicated metadata comment, not a subsection of a license or documentation comment. Decorative block-comment line prefixes such as * are not part of this initial profile.

For example, the existing Swift C-block wrapper can contain:

// swift-tools-version: 6.2

/*
*** YAML START ***
title: Example package
*** YAML END ***
*/

import PackageDescription

Host profiles can use the existing C, HTML, PowerShell, and Lua wrappers where legal. Extending line-comment support to // or other prefixes is a separate adapter decision, not a change to the envelope format.

5. Wrapper representability

Recognize supported comment envelopes for extraction only. Do not enforce host-language comment validity. Provide structured, non-blocking warnings for common hazards such as XML --, HTML comment hazards, and closing tokens within C-style, PowerShell, or Lua payloads. Warnings do not change successful CLI exit status, block reads/writes, or require confirmation.

HTML and XML have different advisory checks. YAML without inner fences can be embedded in an XML comment; payload text containing -- warrants a warning. Do not escape, delete, or convert user data automatically.

6. Discovery, boundaries, and multiplicity

Scan eligible host files for marker candidates regardless of position. Recognize a candidate only under the selected supported wrapper profile and exact logical-line rules. A substring mention in prose is not a marker.

• No recognized markers: metadata is absent; skip payload parsing and do not mutate during discovery.
• Exactly one complete, valid block: return its payload and original source ranges.
• Opening marker without closing marker: structural error.
• Closing marker without opening marker: structural error.
• Nested opening marker, duplicate blocks, or extra recognized markers: structural/multiplicity error.
• Disagreeing format markers (for example, YAML START with TOML END): structural error. This is an already-settled rule, not an open design question.
• Broken comment wrapping or extra non-whitespace content between the host wrapper and marker envelope: structural error.
• Valid envelope with invalid YAML/TOML: payload error.

There MUST be at most one file-level metadata block across both formats. Do not choose the first/last block or merge blocks. Scan sufficiently far to detect duplicates before permitting mutation.

Exact logical marker lines at column one are reserved within the payload, even inside YAML/TOML multiline scalar text. Structural scanning does not depend on first parsing YAML/TOML. There is no marker-escaping mechanism in this initial proposal. An indented marker-looking line is payload content, not a boundary: do not indiscriminately trim leading whitespace. In particular, preserve an indented marker within a YAML block scalar. A column-one marker within a TOML multiline string remains reserved.

Return separate ranges for the host envelope, logical metadata envelope, and payload where useful. These allow subsequent parsers to operate on only the bounded slice while preserving original source locations for diagnostics and mutation.

7. Textual scanning limitation

The initial design uses textual scanning with supported wrapper patterns, not full host-language parsing. Consequently, a string literal or heredoc can contain an exact textual example indistinguishable from a real block to this scanner. This is an acknowledged limitation, not a claim that markers prove comment context.

Document that exact wrapper-qualified marker examples are reserved in participating files. A mere mention of the marker string is harmless if it does not match the full structural-line form. Language-aware lexing may be added later, but is not required by this proposal.

The fixed, distinctive spellings are designed for fast candidate discovery across thousands of files with tools such as ripgrep:

rg -n -F \
  -e '*** YAML START ***' \
  -e '*** YAML END ***' \
  -e '*** TOML START ***' \
  -e '*** TOML END ***' \
  .

Quote the markers and use fixed-string search because asterisks have regex and shell meanings. This yields candidate files and line numbers, not validated blocks or guaranteed matching pairs. The parser still validates complete structural lines, wrappers, formats, and uniqueness. Absence applies only to the files actually searched: hidden-file settings, ignore rules, and binary filtering affect coverage. Discovery tools must document their search scope.

A marker prefilter saves payload parsing, not necessarily file I/O: establishing absence or uniqueness can require a full-file scan.

8. Placement and mutation

Recognition MUST NOT require line one, a fixed distance from a shebang, or EOF. Metadata remains file-level metadata wherever placed. Authors must choose a location where the selected wrapper is legal host syntax.

Updating an existing block MUST preserve its location and host representation unless an explicit conversion or relocation was requested. Preserve all bytes outside the owned host envelope, including license text, declarations, executable content, BOM, and line endings. Removing a block must not remove neighboring license/comments or unrelated blank lines.

Creation placement is a separate policy. Do not infer that successful discovery anywhere means automatic insertion anywhere is safe. A header may follow a required preamble; a footer may need to precede editor settings, signatures, or trailing data. Do not adopt unconditional prepend or append as a universal fallback.

The implementation uses the documented mostly-footer defaults and explicit --before-line N / --after-line N overrides below. These are placement heuristics, not host-language validation. Invalid line boundaries are errors; known host-compatibility hazards are non-blocking warnings.

Do not silently move existing metadata to a preferred header/footer. Keep revision/conflict checks and source-preserving mutation guarantees. Any structural, multiplicity, payload, encoding, or envelope-extraction failure prevents writes to the affected file.

Implementation architecture and decisions

• Add EmbeddedMetadataScanner returning EmbeddedMetadataScan. Optional one-based line hints guide candidate discovery but do not bypass validation or full-file uniqueness checking.
• Establish an EmbeddedMetadataBlock containing declared format, comment representation, source ranges, and physical lines. This establishes matching metadata boundaries and a recognizable wrapper, not host-language or payload validity.
• Retain and adapt separate wrapped-comment and per-line-comment parsers to remove the wrapper/prefixes. Share marker recognition. Pass extracted raw text to the actual YAML/TOML conversion layer for syntax and mapping validation.
• Rename unreleased non-Markdown-specific public APIs directly to embedded-metadata terminology; retain the shared FrontMatter data model. No compatibility aliases or migration.
• Use Parsing for text parsing and Swift Testing for tests.
• Add mutually exclusive --before-line N / --after-line N on creation-capable commands. These select boundaries in the original source, not authorization. Reject invalid positions, use on Markdown, and relocation of an existing block through creation options.
• Default mostly to a footer. Insert before recognized Ruby __END__, PowerShell signatures, and trailing Emacs/Vim settings, choosing the earliest applicable anchor. These are helpful heuristics, not host-language safety guarantees. Preserve leading declarations and licenses through footer placement.
• Rename the unreleased hash override to --hash-comment-metadata. Keep fm, existing format/creation options, and other selection behavior.
• Map Python/Pyi to hash comments; keep Swift C-block comments. Use XML/XHTML/SVG-specific advisory profiles distinct from HTML.
• Integrate the scanner/extractors with CLI and record/rules analysis. Keep structural/payload errors blocking, compatibility warnings non-blocking, and revision-checked source-preserving writes.
• Update CLI help, standalone spec, DocC, examples, and synchronized skill copies.

Acceptance criteria

• Document the application-independent envelope, terminology, and host profiles as a standalone specification other tools can implement.
• Read and mutate marked YAML and TOML blocks with exact format-specific START/END markers and no inner fences.
• Accept empty and whitespace-only lines at the beginning/end of and within the payload; preserve significant indentation and multiline-scalar whitespace.
• Cover LF-only recognition, CRLF non-recognition, UTF-8 BOM, and a final line without newline, preserving source outside the owned envelope.
• Discover a single block at beginning, middle, or end without shebang-relative recognition rules.
• Treat unmarked non-Markdown frontmatter as absent; traditional Markdown frontmatter remains unchanged.
• Diagnose orphan/missing markers, nesting, duplicates across formats, disagreeing format markers, malformed wrappers, extra surrounding content, and invalid payloads without writing.
• Scan for ambiguity before any mutation and report original source locations.
• Support hash-comment and dedicated block-comment representations, including whitespace-only physical lines and exact prefix removal.
• Use hash comments for Python/Pyi; fixtures preserve module docstrings, future imports, shebangs, and encoding declarations and pass Python compilation after mutation.
• A Package.swift fixture retains its leading tools-version declaration after metadata creation/update.
• Warn on known host-comment hazards without blocking valid metadata reads/writes; cover closing tokens and HTML/XML differences, including XML --.
• Explicit placement or documented supported automatic placement handles required preambles; footer defaults and explicit line overrides behave as documented, without claiming host-language validation.
• Updating/removing metadata preserves unrelated source and does not move blocks or remove neighboring comments.
• Verify logical-column-one marker recognition, trailing whitespace tolerance, preservation of indented marker-like scalar content, and single-document YAML semantics.
• Document fast fixed-string discovery with candidate line numbers and explicit search-scope limitations.
• Document and test the textual-scanner limitation with exact marker examples inside host strings.
• Update CLI help, project documentation, and synchronized skill copies; run the relevant Swift tests and required package checks.

Related issues

• Use semantically safe inline frontmatter for Python files #133: Python representation and semantic safety. This proposal retains its semantic goals but explicitly does not require its earlier migration proposal.
• Preserve leading XML declarations when creating frontmatter #132: XML declaration placement. This proposal retains declaration preservation and additionally requires XML-valid comment contents; its older fenced YAML-in-XML example should be replaced by the unfenced format-specific markers above.

Related issues are not automatically closed or edited by this proposal.