Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ unexpected_cfgs = { level = "warn", check-cfg = ['cfg(loom)'] }
serde = { version = "^1", features = ["derive"] }
serde_json = "^1"
serde_yaml = "^0.9"
# Stable SHA-256 record ids for canonical Filament extraction (FR-045).
sha2 = "^0.10"
indexmap = { version = "^2", features = ["serde"] }
thiserror = "^2"
# Regex driver for the Query API (FR-010). Patterns are author-side
Expand All @@ -51,6 +53,8 @@ uuid = { version = "^1", features = ["v7", "serde"] }
# forward-compatible wheel (NFR-016). `extension-module` drops libpython
# linkage so the wheel loads in any host interpreter.
pyo3 = { version = "0.29", features = ["extension-module", "abi3-py39"], optional = true }
# wasm entropy source for `uuid` v7 under `wasm32-unknown-unknown` (see `wasm` feature).
getrandom = { version = "^0.3", features = ["wasm_js"], optional = true }

[dev-dependencies]
proptest = "^1"
Expand All @@ -77,7 +81,7 @@ resolve-file = ["jsonschema/resolve-file"]
# / `from_default` remain available but operate against whatever
# filesystem the wasm host exposes (typically none under
# `--target web`).
wasm = []
wasm = ["uuid/js", "dep:getrandom"]

[[bench]]
name = "parse"
Expand Down
52 changes: 52 additions & 0 deletions reviews/2026-07-11-filament-core-extraction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
---
id: SR-001
title: "Gap analysis — canonical Filament core-data extraction (FR-045/FR-046/NFR-020)"
type: SpecReview
analysis: gap-analysis
scope: "spec/functional/FR-045-filament-core-extraction-engine.md, spec/functional/FR-046-filament-extraction-bindings.md, spec/non-functional/NFR-020-filament-extraction-boundary.md, spec/usecase/US-016-canonical-filament-extraction.md, spec/functional/FR-006-frontmatter-with-fallback.md, spec/tests.md"
review_set: subset
relationships:
- { target: "ix://agent-ix/quire-rs/spec/functional/FR-045", type: reviews }
- { target: "ix://agent-ix/quire-rs/spec/functional/FR-046", type: references }
- { target: "ix://agent-ix/quire-rs/spec/tests", type: references }
---

## Summary

Code-review + gap-analysis of the **rebased-onto-main** canonical Filament extraction
feature (originally reviewed on a stale branch under FR-040/FR-041; re-verified here under
the landed IDs FR-045/FR-046/US-016/NFR-020 + FR-006-AC-7). The engine core is well
covered; review surfaced that several binding/static Test Cases were marked ✅ without
in-repo backing — feasible ones were backed, the rest honestly re-marked.

## Verdict

**CONDITIONAL** — no incorrect behavior; the engine core (FR-045) is backed by 20 unit +
fixture tests. All matrix over-claims found in review were resolved: two were genuinely
backed but untagged (fixed), two gained real in-repo tests (fixed), and three
downstream/CI/inspection TCs were honestly downgraded to 🚧 to match main's convention.

## Findings

| ID | Severity | Summary | Refs |
| ------- | -------- | ------------------------------------------------------------------------------------------------------------ | ----------------------------- |
| FND-001 | medium | RESOLVED — TC-703/704 (corpus isolation + determinism) were backed by the harness tests but carried no tracking tag; added `// TC-703`/`// TC-704` tags. | TC-703, TC-704 |
| FND-002 | medium | RESOLVED — TC-686 (Python binding parity) and TC-690 (NFR-020-AC-1 static boundary) had no in-repo test; added `tests/python/test_bindings.py` filament cases (TC-686, CI-run) and `tests/filament_boundary_audit.rs` (TC-690). | TC-686, TC-690, NFR-020-AC-1 |
| FND-003 | low | ACCEPTED — TC-687 (`@agent-ix/quire-wasm` exports), TC-688 (binding no-policy inspection), TC-689 (wasm-target compile) have no in-repo test; verified downstream / by CI wasm-target / by inspection. Downgraded ✅→🚧 with method notes to match main's convention for Static/Compile TCs. | TC-687, TC-688, TC-689 |
| FND-004 | low | NOTE — a pre-existing `useless_borrows_in_formatting` failure in `tests/parser_real_documents.rs` (surfaced by CI's toolchain bump to rust 1.97) was fixed incidentally so the gate passes; not part of the feature. | tests/parser_real_documents.rs |

## Coverage

- **Engine (FR-045):** TC-681..685 (unit), TC-691..702 + TC-705 (fixtures), TC-703/704
(harness isolation + determinism), TC-706 (FR-006-AC-7 frontmatter status) — all backed
by tagged tests, verified under rust 1.97 (387 tests, clippy `-D warnings`, fmt clean).
- **Bindings (FR-046):** TC-686 backed by a new Python binding test (CI `python.yml`);
TC-687/688/689 verified downstream/CI/inspection (🚧).
- **Boundary (NFR-020):** TC-690 backed by a new static-inspection test; TC-704 (determinism)
and TC-686 (native-value parity) backed.
- **Reverse gap:** 0 untraced code — all new code owns a requirement (FR-045/FR-046/NFR-020,
FR-006-AC-7 via CR-011). No stale FR-040/FR-041/US-015/CR-010/TC-63x IDs leaked into code
or the new spec files (verified by diff scan).
- **Semantic review:** not run (mechanical gates only); engine edge-harvesting confirmed
architecturally independent of main's FR-040 object-edge vocabulary (standalone boundary,
no registry).
2 changes: 2 additions & 0 deletions spec/functional/FR-006-frontmatter-with-fallback.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ relationships:
2. If `markdown` begins with `---\n` but no closing `---\n` line exists, return `FrontmatterResult { frontmatter: None, body: markdown.into() }`.
3. If a closing `---\n` exists but the content between the fences is not valid YAML, return `FrontmatterResult { frontmatter: None, body: markdown.into() }` — **the entire input is body**. This matches the TS/Py reference: malformed frontmatter is NOT an error; it is treated as content.
4. If the content is valid YAML, return `FrontmatterResult { frontmatter: Some(parsed), body: text_after_closing_fence }`.
5. **Frontmatter status (CR-011).** The result SHALL additionally carry a typed `status ∈ {Absent, Present, Malformed}` reporting *why* `frontmatter` is `None`, or that it parsed: branches 1–2 yield `Absent` (no block, or an unterminated fence); an invalid-YAML block (branch 3) or a valid-but-non-mapping value (array/scalar/bool/number) yields `Malformed`; an **empty / whitespace / comment-only** block (YAML null) yields `Absent` (it carries no metadata and is indistinguishable from having no frontmatter — e.g. a `---`…`---` pair of thematic breaks — so it is not a parse failure); and branch 4 yields `Present`. This is a parity-preserving extension: the `frontmatter` and `body` outputs are byte-identical across all branches (like the BOM/CRLF extensions), so the TS/Py reference comparison is unaffected. The status exists so a boundary consumer (e.g. the Filament extraction engine, [FR-045](./FR-045-filament-core-extraction-engine.md)) can distinguish an absent block from a malformed one without re-deriving it from the raw markdown.

The parsed frontmatter is a `serde_json::Map<String, Value>` (JSON-compatible value tree), not a typed struct — typed deserialization is the consumer's responsibility.

Expand All @@ -33,6 +34,7 @@ The parsed frontmatter is a `serde_json::Map<String, Value>` (JSON-compatible va
| FR-006-AC-4 | [FR-001](./FR-001-render-dispatch.md) returns `frontmatter: None, body: full original input`. | Test |
| FR-006-AC-5 | [FR-001](./FR-001-render-dispatch.md) (BOM-prefixed input) returns the same result as the BOM-free equivalent — frontmatter parsed, body equal to `"body"` (no BOM in body). | Test |
| FR-006-AC-6 | `extract_frontmatter("\u{FEFF}# heading")` (BOM-prefixed, no frontmatter) returns `frontmatter: None, body: "# heading"` (BOM stripped from body). | Test |
| FR-006-AC-7 | The result `status` (CR-011) classifies each branch: absent/unterminated → `Absent`; a valid mapping → `Present`; invalid YAML or a non-null non-mapping value (array/scalar) → `Malformed`; an empty/whitespace/comment-only block (YAML null) → `Absent` (not a parse failure). | Test (TC-706) |

## Dependencies

Expand Down
97 changes: 97 additions & 0 deletions spec/functional/FR-045-filament-core-extraction-engine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
id: FR-045
title: "Canonical Filament core extraction engine"
type: FR
relationships:
- target: "ix://agent-ix/quire-rs/spec/usecase/US-016"
type: "implements"
- target: "ix://agent-ix/quire-rs/spec/functional/FR-011"
type: "requires"
- target: "ix://agent-ix/quire-rs/spec/functional/FR-026"
type: "requires"
---
# [FR-045] Canonical Filament core extraction engine

## Description

The `quire-rs` library SHALL provide a Filament-facing extraction API that accepts one
markdown document plus caller-provided ObjectType snapshots and returns a JSON-serializable
core extraction result compatible with the shared `CoreExtractionResult` contract.

## Inputs

- Project id, document id, optional artifact id, relative path, repository name, and markdown text
- A caller-provided ObjectType snapshot containing name, schema, allowed links,
optional `body_extraction`, and plugin flag fields

## Outputs

- Object type records
- Graph node records
- Graph edge records
- Extraction diagnostics
- Extraction errors

## Behavior

- The engine SHALL parse frontmatter and markdown using the existing Rust parser.
- When a document has no usable frontmatter block (absent, an unterminated opening fence,
or an empty / whitespace / comment-only block), the engine SHALL return an empty
extraction result with a `no_frontmatter` informational diagnostic and no extraction error.
- When a document has a complete frontmatter fence block that does not parse into a YAML
mapping (invalid YAML, or a non-empty non-mapping value such as an array or scalar), the
engine SHALL return an empty extraction result with a `parse_failed` extraction error and
a `frontmatter_unparsable` error-severity diagnostic, so the failure reaches Filament's
per-document error index.
- When `frontmatter.object` names an unknown ObjectType, the engine SHALL return no node
for that object.
- When `frontmatter.object` names an unknown ObjectType, the engine SHALL emit an
`unknown_object_type` diagnostic.
- When the engine receives an ObjectType with `body_extraction` absent and `has_plugin`
false, the engine SHALL run Tier 1 frontmatter extraction.
- When an ObjectType has `body_extraction` and `has_plugin` is false, the engine SHALL
evaluate the existing Rust body-extraction DSL and validate each emitted record.
- When an ObjectType has `has_plugin` true, the engine SHALL surface an extraction error
or diagnostic and SHALL NOT execute Python plugin discovery.
- The engine SHALL harvest frontmatter relationship sugar and body `ix://` markdown links
into graph edges with stable provenance metadata.
- The engine SHALL deduplicate duplicate graph edges by source, edge type, and target,
preserving the first edge and emitting a diagnostic for duplicates.
- The engine SHALL produce stable SHA-256 ids for object type, node, and edge records.

## Constraints

| ID | Constraint | Type | Validation |
|----|------------|------|------------|
| FR-045-CON-1 | The extraction engine SHALL NOT read PGlite, Electron IPC, HTTP, auth, CloudManager sync, workspace watch queues, or embeddings. | Architecture | Inspection |
| FR-045-CON-2 | ObjectType snapshots SHALL be supplied by the caller. | Architecture | Test |
| FR-045-CON-3 | The engine SHALL NOT discover a runtime registry over the network or from service configuration. | Architecture | Test |
| FR-045-CON-4 | Non-`ix://` graph references SHALL normalize to `ix://agent-ix/<repo_name>/<value>` before record id generation. | Compatibility | Test |

## Acceptance Criteria

| ID | Criteria | Verification |
|----|----------|--------------|
| FR-045-AC-1 | A Tier 1 fixture emits one validated graph node whose `dataJson` includes frontmatter `id` as `code` and frontmatter `title` as `title`. | Test (TC-681) |
| FR-045-AC-2 | A Tier 2 fixture emits graph nodes and record-derived graph edges equivalent to the existing Rust DSL extraction result for the same ObjectType snapshot. | Test (TC-682) |
| FR-045-AC-3 | Unknown ObjectType, no-frontmatter, malformed `ix://`, duplicate-edge, and plugin-flag fixtures produce diagnostics or errors without panicking. | Test (TC-683) |
| FR-045-AC-4 | Frontmatter relationship sugar and body `ix://` links produce deterministic graph edges with source/original-target or original-uri metadata. | Test (TC-684) |
| FR-045-AC-5 | Repeating extraction with identical input produces byte-identical JSON output ordering and stable record ids. | Test (TC-685) |
| FR-045-AC-6 | A document with a complete but unparsable frontmatter fence block yields a non-empty `errors` entry (`parse_failed`) and a `frontmatter_unparsable` diagnostic, while a document with no frontmatter block stays clean (`no_frontmatter` diagnostic, empty `errors`). | Test (TC-705) |

> **CR-011 note:** The original behavior ("frontmatter absent **or unparsable** → empty
> result with a `no_frontmatter` informational diagnostic") conflated two distinct cases
> and was corrected per issue #127: a *malformed* frontmatter block was silently treated
> as a clean extraction, so per-document parse failures never reached Filament's
> `index_errors` (the `MarkStale → apply_stale` path only records a document with a
> non-empty `errors` list). The engine now distinguishes **absent** (clean skip) from
> **present-but-unparsable** (extraction error). The absent-vs-malformed classification is
> owned by the parser — [FR-006](./FR-006-frontmatter-with-fallback.md) now exposes a typed
> frontmatter `status` — and this engine reads that status rather than re-deriving it, so
> there is a single source of truth. Parser FR-006 parity (frontmatter/body outputs) is
> unchanged; the malformed → error decision is Filament-boundary policy.

## Dependencies

- **Upstream**: [FR-011](./FR-011-body-extraction-dsl.md), [FR-026](./FR-026-intra-spec-reference-resolution.md), [FR-028](./FR-028-expanded-python-binding-surface.md)
- **Downstream**: [FR-046](./FR-046-filament-extraction-bindings.md), `filament-parser-lib` compatibility shim, and Filament IDE worker sync
Loading
Loading