Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ unexpected_cfgs = { level = "warn", check-cfg = ['cfg(loom)'] }
serde = { version = "^1", features = ["derive"] }
serde_json = "^1"
serde_yaml = "^0.9"
sha2 = "^0.10"
indexmap = { version = "^2", features = ["serde"] }
thiserror = "^2"
# Regex driver for the Query API (FR-010). Patterns are author-side
Expand All @@ -51,6 +52,7 @@ uuid = { version = "^1", features = ["v7", "serde"] }
# forward-compatible wheel (NFR-016). `extension-module` drops libpython
# linkage so the wheel loads in any host interpreter.
pyo3 = { version = "0.29", features = ["extension-module", "abi3-py39"], optional = true }
getrandom = { version = "^0.3", features = ["wasm_js"], optional = true }

[dev-dependencies]
proptest = "^1"
Expand All @@ -77,7 +79,7 @@ resolve-file = ["jsonschema/resolve-file"]
# / `from_default` remain available but operate against whatever
# filesystem the wasm host exposes (typically none under
# `--target web`).
wasm = []
wasm = ["uuid/js", "dep:getrandom"]

[[bench]]
name = "parse"
Expand Down
60 changes: 60 additions & 0 deletions reviews/2026-07-11-issue-127-malformed-frontmatter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
id: SR-001
title: "Gap analysis — issue #127 malformed-frontmatter parse failure"
type: SpecReview
analysis: gap-analysis
scope: "spec/functional/FR-040-filament-core-extraction-engine.md, spec/functional/FR-006-frontmatter-with-fallback.md, spec/tests.md"
review_set: subset
relationships:
- { target: "ix://agent-ix/quire-rs/spec/functional/FR-040", type: reviews }
- { target: "ix://agent-ix/quire-rs/spec/functional/FR-006", type: references }
- { target: "ix://agent-ix/quire-rs/spec/tests", type: references }
---

## Summary

Post-implementation gap analysis of the issue #127 change, which makes the Filament
extraction engine emit a `parse_failed` error when a document has a complete-but-unparsable
frontmatter fence block (so the failure reaches Filament's `index_errors` via
`MarkStale → apply_stale`). The change spans the parser (a new typed `FrontmatterStatus` on
`extract_frontmatter`, FR-006/CR-010) and the Filament boundary (`FR-040-AC-6`). Mechanical
gates (matrix backing, reverse-gap) pass; two low-severity process observations are recorded.
The optional semantic review was skipped at the user's request.

## Verdict

**CONDITIONAL** — no incomplete tasks, no unbacked matrix Test Cases, and no high/medium
findings. FND-002 (traceability) has been **resolved** post-review; FND-001 remains as an
accepted, non-blocking process note (no plan bundle for a goal-driven bugfix, consistent with
how FR-040/FR-041 themselves landed).

## Findings

| ID | Severity | Summary | Refs |
| ------- | -------- | --------------------------------------------------------------------------------------------------------- | --------------------------------- |
| FND-001 | low | The #127 change has no owning `plan/tasks/NNN` bundle entry (goal-driven fix); FR-040/FR-041 also landed without plan-task tracking. Accepted — change is complete and verified regardless. | plan/tasks/README.md, FR-040 |
| FND-002 | low | RESOLVED — the CR-010 empty/null→`Absent` classification now has a dedicated `FR-006-AC-7 → TC-658`, and the parser test carries the `FR-006-AC-7 / TC-658` tracking tag. Coverage updated 340 → 341. | FR-006-AC-7, TC-658 |

## Coverage

- **Tasks done:** N/A — no formal `plan/tasks/` bundle owns FR-040/FR-041 or the #127 fix
(see FND-001). No task is incomplete or blocked.
- **Matrix Test Cases backed by tagged test:** 2 / 2 new rows. `TC-657` is backed by the
`tc657_malformed_frontmatter_is_a_parse_failure_but_absent_is_clean` unit test
(`src/filament.rs`) and the `["TC-657"]`-tagged `frontmatter-unparsable-error` fixture in
`tests/fixtures/filament_core/graph_cases.json`, exercised by the corpus harness (value +
determinism). `TC-658` (added to resolve FND-002) is backed by the
`empty_or_comment_only_frontmatter_is_absent_not_malformed` parser test carrying the
`FR-006-AC-7 / TC-658` tag. Both `FR-040-AC-6 → TC-657` and `FR-006-AC-7 → TC-658` mappings
are present in the AC→TC audit table; the coverage count is consistently updated
(339 → 341, integrity note aligned). No existing matrix row was disturbed.
- **Untraced behaviors / stubs:** 0. All new code has an owning requirement — the
`FrontmatterStatus` enum and `status` field trace to FR-006 §5 (CR-010); the
`malformed_frontmatter_result` / `absent_frontmatter_result` helpers and the
`parse_failed` / `frontmatter_unparsable` emission trace to FR-040-AC-6 and the split
FR-040 behavior bullet. The two incidental clippy fixes (`Tier2Extraction` type alias,
`stable_id` borrow) are behavior-preserving and carry no spec surface.
- **Semantic review:** skipped (user opted out). Intent↔test↔code for the FR-040-AC-6 triple
was nonetheless spot-checked during code-review: `TC-657` asserts the exact AC behavior
(empty nodes/edges, `parse_failed` in `errors`, `frontmatter_unparsable` error-severity
diagnostic; absent stays clean) against the real `extract_filament_core` path (no mocks).
2 changes: 2 additions & 0 deletions spec/functional/FR-006-frontmatter-with-fallback.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ relationships:
2. If `markdown` begins with `---\n` but no closing `---\n` line exists, return `FrontmatterResult { frontmatter: None, body: markdown.into() }`.
3. If a closing `---\n` exists but the content between the fences is not valid YAML, return `FrontmatterResult { frontmatter: None, body: markdown.into() }` — **the entire input is body**. This matches the TS/Py reference: malformed frontmatter is NOT an error; it is treated as content.
4. If the content is valid YAML, return `FrontmatterResult { frontmatter: Some(parsed), body: text_after_closing_fence }`.
5. **Frontmatter status (CR-010).** The result SHALL additionally carry a typed `status ∈ {Absent, Present, Malformed}` that reports *why* `frontmatter` is `None`, or that it parsed: branches 1–2 yield `Absent` (no block, or an unterminated fence); an invalid-YAML block (branch 3) or a valid-but-non-mapping value (array/scalar/bool/number) yields `Malformed`; an **empty / whitespace / comment-only** block (YAML null) yields `Absent` (it carries no metadata and is indistinguishable from having no frontmatter — e.g. a `---`…`---` pair of thematic breaks — so it is not a parse failure); and branch 4 yields `Present`. This is a parity-preserving extension: the `frontmatter` and `body` outputs are byte-identical across all branches (like the BOM/CRLF extensions), so the TS/Py reference comparison is unaffected. The status exists so a boundary consumer (e.g. the Filament extraction engine, [FR-040](./FR-040-filament-core-extraction-engine.md)) can distinguish an absent block from a malformed one without re-deriving it from the raw markdown.

The parsed frontmatter is a `serde_json::Map<String, Value>` (JSON-compatible value tree), not a typed struct — typed deserialization is the consumer's responsibility.

Expand All @@ -33,6 +34,7 @@ The parsed frontmatter is a `serde_json::Map<String, Value>` (JSON-compatible va
| FR-006-AC-4 | [FR-001](./FR-001-render-dispatch.md) returns `frontmatter: None, body: full original input`. | Test |
| FR-006-AC-5 | [FR-001](./FR-001-render-dispatch.md) (BOM-prefixed input) returns the same result as the BOM-free equivalent — frontmatter parsed, body equal to `"body"` (no BOM in body). | Test |
| FR-006-AC-6 | `extract_frontmatter("\u{FEFF}# heading")` (BOM-prefixed, no frontmatter) returns `frontmatter: None, body: "# heading"` (BOM stripped from body). | Test |
| FR-006-AC-7 | The result `status` (CR-010) classifies each branch: absent/unterminated → `Absent`; a valid mapping → `Present`; invalid YAML or a non-null non-mapping value (array/scalar) → `Malformed`; an empty/whitespace/comment-only block (YAML null) → `Absent` (not a parse failure). | Test (TC-658) |

## Dependencies

Expand Down
97 changes: 97 additions & 0 deletions spec/functional/FR-040-filament-core-extraction-engine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
id: FR-040
title: "Canonical Filament core extraction engine"
type: FR
relationships:
- target: "ix://agent-ix/quire-rs/spec/usecase/US-015"
type: "implements"
- target: "ix://agent-ix/quire-rs/spec/functional/FR-011"
type: "requires"
- target: "ix://agent-ix/quire-rs/spec/functional/FR-026"
type: "requires"
---
# [FR-040] Canonical Filament core extraction engine

## Description

The `quire-rs` library SHALL provide a Filament-facing extraction API that accepts one
markdown document plus caller-provided ObjectType snapshots and returns a JSON-serializable
core extraction result compatible with the shared `CoreExtractionResult` contract.

## Inputs

- Project id, document id, optional artifact id, relative path, repository name, and markdown text
- A caller-provided ObjectType snapshot containing name, schema, allowed links,
optional `body_extraction`, and plugin flag fields

## Outputs

- Object type records
- Graph node records
- Graph edge records
- Extraction diagnostics
- Extraction errors

## Behavior

- The engine SHALL parse frontmatter and markdown using the existing Rust parser.
- When a document has no usable frontmatter block (absent, an unterminated opening fence,
or an empty / whitespace / comment-only block), the engine SHALL return an empty
extraction result with a `no_frontmatter` informational diagnostic and no extraction error.
- When a document has a complete frontmatter fence block that does not parse into a YAML
mapping (invalid YAML, or a non-empty non-mapping value such as an array or scalar), the
engine SHALL return an empty extraction result with a `parse_failed` extraction error and
a `frontmatter_unparsable` error-severity diagnostic, so the failure reaches Filament's
per-document error index.
- When `frontmatter.object` names an unknown ObjectType, the engine SHALL return no node
for that object.
- When `frontmatter.object` names an unknown ObjectType, the engine SHALL emit an
`unknown_object_type` diagnostic.
- When the engine receives an ObjectType with `body_extraction` absent and `has_plugin`
false, the engine SHALL run Tier 1 frontmatter extraction.
- When an ObjectType has `body_extraction` and `has_plugin` is false, the engine SHALL
evaluate the existing Rust body-extraction DSL and validate each emitted record.
- When an ObjectType has `has_plugin` true, the engine SHALL surface an extraction error
or diagnostic and SHALL NOT execute Python plugin discovery.
- The engine SHALL harvest frontmatter relationship sugar and body `ix://` markdown links
into graph edges with stable provenance metadata.
- The engine SHALL deduplicate duplicate graph edges by source, edge type, and target,
preserving the first edge and emitting a diagnostic for duplicates.
- The engine SHALL produce stable SHA-256 ids for object type, node, and edge records.

## Constraints

| ID | Constraint | Type | Validation |
|----|------------|------|------------|
| FR-040-CON-1 | The extraction engine SHALL NOT read PGlite, Electron IPC, HTTP, auth, CloudManager sync, workspace watch queues, or embeddings. | Architecture | Inspection |
| FR-040-CON-2 | ObjectType snapshots SHALL be supplied by the caller. | Architecture | Test |
| FR-040-CON-3 | The engine SHALL NOT discover a runtime registry over the network or from service configuration. | Architecture | Test |
| FR-040-CON-4 | Non-`ix://` graph references SHALL normalize to `ix://agent-ix/<repo_name>/<value>` before record id generation. | Compatibility | Test |

## Acceptance Criteria

| ID | Criteria | Verification |
|----|----------|--------------|
| FR-040-AC-1 | A Tier 1 fixture emits one validated graph node whose `dataJson` includes frontmatter `id` as `code` and frontmatter `title` as `title`. | Test (TC-633) |
| FR-040-AC-2 | A Tier 2 fixture emits graph nodes and record-derived graph edges equivalent to the existing Rust DSL extraction result for the same ObjectType snapshot. | Test (TC-634) |
| FR-040-AC-3 | Unknown ObjectType, no-frontmatter, malformed `ix://`, duplicate-edge, and plugin-flag fixtures produce diagnostics or errors without panicking. | Test (TC-635) |
| FR-040-AC-4 | Frontmatter relationship sugar and body `ix://` links produce deterministic graph edges with source/original-target or original-uri metadata. | Test (TC-636) |
| FR-040-AC-5 | Repeating extraction with identical input produces byte-identical JSON output ordering and stable record ids. | Test (TC-637) |
| FR-040-AC-6 | A document with a complete but unparsable frontmatter fence block yields a non-empty `errors` entry (`parse_failed`) and a `frontmatter_unparsable` diagnostic, while a document with no frontmatter block stays clean (`no_frontmatter` diagnostic, empty `errors`). | Test (TC-657) |

> **CR-010 note:** The original behavior ("frontmatter absent **or unparsable** → empty
> result with a `no_frontmatter` informational diagnostic") conflated two distinct cases
> and was corrected per issue #127: a *malformed* frontmatter block was silently treated
> as a clean extraction, so per-document parse failures never reached Filament's
> `index_errors` (the `MarkStale → apply_stale` path only records a document with a
> non-empty `errors` list). The engine now distinguishes **absent** (clean skip) from
> **present-but-unparsable** (extraction error). The absent-vs-malformed classification is
> owned by the parser — [FR-006](./FR-006-frontmatter-with-fallback.md) now exposes a typed
> frontmatter `status` — and this engine reads that status rather than re-deriving it, so
> there is a single source of truth. Parser FR-006 parity (frontmatter/body outputs) is
> unchanged; the malformed → error decision is Filament-boundary policy.

## Dependencies

- **Upstream**: [FR-011](./FR-011-body-extraction-dsl.md), [FR-026](./FR-026-intra-spec-reference-resolution.md), [FR-028](./FR-028-expanded-python-binding-surface.md)
- **Downstream**: [FR-041](./FR-041-filament-extraction-bindings.md), `filament-parser-lib` compatibility shim, and Filament IDE worker sync
Loading
Loading