spec: codify V7 demo QA and audit

Author: geraldarthurdavis

VULNERABILITY_AUDIT_2026-04-02.md

@@ -1,137 +1,188 @@
-# ENGI Demo Checklist
+# ENGI Demo Checklist — Spec V7
 
-Use this before presenting the demo.
+Use this before an interactive V7 demo, QA pass, or live walkthrough.
 
 ---
 
-## Pre-demo setup
+## 0. Server / environment
 
-- [ ] Open the demo at `http://127.0.0.1:4318`
-- [ ] Confirm the page loads successfully
-- [ ] Confirm the thesis strip is visible
-- [ ] Confirm the top summary cards render
-- [ ] Set mode appropriately:
-  - [ ] `light` for live presentation
-  - [ ] `heavy` only if deeper inspection is expected
-- [ ] Click **Reset demo** before starting
-- [ ] Confirm seeded state appears clean:
-  - [ ] 3 assets
-  - [ ] 0 public receipts
-  - [ ] proof group summary shows clean initial state
+- [ ] Start the demo:
+
+```bash
+cd /Users/garrettmaring/Developer/ENGI/engi-demo
+npm start
+```
+
+- [ ] Open: `http://127.0.0.1:4318`
+- [ ] Confirm the page loads
+- [ ] Confirm the hero says **Spec V7**
+- [ ] Confirm the primary CTA is **Make ENGI branch**
+- [ ] Confirm no stale V6/query-era language is visible in the main flow
 
 ---
 
-## Seeded scenario sanity check
+## 1. Clean seeded state
+
+- [ ] Click **Reset demo** before starting
+- [ ] Confirm seeded state is clean:
+  - [ ] 3 assets
+  - [ ] 1 need scenario
+  - [ ] no latest run yet
+- [ ] Confirm the active conformance profile shows **Profile A**
+- [ ] Confirm the seeded scenario is the auth rollback case
 
-- [ ] Buyer is `Frontier Code Systems`
-- [ ] Seeded query is `enterprise auth migration rollback for monorepo services with issuer mismatch`
-- [ ] Expected top asset is `Enterprise auth migration rollback playbook`
-- [ ] Benchmark name is `production-auth-remediation`
+Expected seeded scenario cues:
+- buyer: `Frontier Code Systems`
+- repo: `frontier/demo-auth`
+- branch: `ENGI-auth-issuer-rollback`
+- benchmark workflow / GitHub Actions evidence present
 
 ---
 
-## Gold-path presentation checklist
-
-### 1. Openly writable
-- [ ] Point to thesis strip
-- [ ] Show `1. Public deposits`
-- [ ] Explain that public commitments are visible while corpus payloads stay sealed
-
-### 2. Measurably readable
-- [ ] Run the licensed query or click guided demo
-- [ ] Show the private bundle
-- [ ] Explain that the system assembles a need-matched bundle rather than revealing the full corpus
-
-### 3. Provable
-- [ ] In light mode, mention the collapsed proof summary
-- [ ] If needed, expand proof group
-- [ ] Show at least:
-  - [ ] ranking explanation
-  - [ ] conservation check
-  - [ ] public receipts
-- [ ] Mention policy/provenance if asked
-
-### 4. Valuable
-- [ ] End on the benchmark panel
-- [ ] Explain baseline outcome
-- [ ] Explain treatment outcome
-- [ ] Explain why the licensed bundle helped
-- [ ] Land the buyer impact clearly
+## 2. V7 gold-path interactive run
+
+### A. Need measurement
+
+- [ ] Show that the need is GitHub-bound, not a free-text-only query
+- [ ] Point out:
+  - [ ] parser contract
+  - [ ] canonical run evidence
+  - [ ] field derivations / normalization closure
+  - [ ] conformance profile labels
+
+What should be visible after the run:
+- [ ] measured `needId`
+- [ ] parser fail-closed framing
+- [ ] need derivation / provenance surface
+
+### B. Candidate recall + ranking
+
+- [ ] Run **Make ENGI branch**
+- [ ] Confirm evaluated candidates render
+- [ ] Confirm the top asset is still the rollback/auth material
+- [ ] Show:
+  - [ ] multi-channel recall / fusion
+  - [ ] need match
+  - [ ] benchmark impact
+  - [ ] actionability
+  - [ ] penalties
+  - [ ] whole-asset score / explainability
+
+### C. Verification + use tiers
+
+- [ ] Confirm verification is separate from ranking
+- [ ] Show:
+  - [ ] issuance verification
+  - [ ] provenance verification
+  - [ ] verification sufficiency
+  - [ ] issuer policy status
+- [ ] Confirm use tiers are visible
+- [ ] Confirm branch-mode rights are visible / inferable
+
+### D. Asset pack + branch artifacts
+
+- [ ] Confirm an `assetPackId` is generated
+- [ ] Confirm branch artifacts include at least:
+  - [ ] `.engi/need.json`
+  - [ ] `.engi/verification-report.json`
+  - [ ] `.engi/asset-pack.lock.json`
+  - [ ] `.engi/authorization-decisions.json`
+  - [ ] `.engi/sensitive-data-flow.json`
+  - [ ] `.engi/policy-release.json`
+  - [ ] `.engi/unit-catalog.json`
+  - [ ] `.engi/pipeline-telemetry.json`
+  - [ ] `.engi/system-proof-bundle.json`
+  - [ ] `ENGI_NEED.md`
+
+### E. Settlement / proof closure
+
+- [ ] Confirm settlement preview exists
+- [ ] Confirm journal diff exists
+- [ ] Confirm `debitsEqualCredits = true`
+- [ ] Confirm asset-pack-lock binding is visible in settlement/proof surfaces
+- [ ] Confirm participating assets and shares are visible
 
 ---
 
-## Expected live outputs after gold path
+## 3. Advanced V7 surfaces
 
-After one normal gold-path run, expect roughly:
+### Content-unit semantics / vector interfaces
 
-- [ ] private bundle issued successfully
-- [ ] top-ranked asset: `Enterprise auth migration rollback playbook`
-- [ ] allocation conservation: `verified`
-- [ ] public receipts present
-- [ ] benchmark uplift: `2500 bp`
-- [ ] buyer impact mentions:
-  - [ ] higher-confidence incident recovery
-  - [ ] faster safe rollback
-  - [ ] lower risk of breaking live sessions
+- [ ] Confirm unit catalog exists
+- [ ] Confirm content units expose:
+  - [ ] semantic summary
+  - [ ] embedding / vector contract metadata
+  - [ ] provenance / unit hash
+- [ ] Confirm these are framed as **Profile A deterministic stand-ins** for richer Profile B embeddings
 
----
+### Static vs inferred evaluator boundaries
+
+- [ ] Confirm eval/proof surfaces distinguish between:
+  - [ ] static analysis / deterministic measurement
+  - [ ] inferred measurement / evaluator surfaces
+  - [ ] stand-in vs production-boundary behavior
 
-## If presenting in light mode
+### Telemetry / harness
 
-- [ ] Keep proof group collapsed until the proof step
-- [ ] Use the proof summary card as the trust-at-a-glance surface
-- [ ] Do not get stuck inside raw proof payloads
-- [ ] Return quickly to the benchmark ending
+- [ ] Confirm pipeline telemetry artifact exists
+- [ ] Confirm telemetry explains:
+  - [ ] unit extraction
+  - [ ] recall channels / fusion
+  - [ ] score composition
+  - [ ] verification decisions
+  - [ ] use-tier propagation
+  - [ ] artifact materialization
+  - [ ] settlement / shares
+  - [ ] fail-closed behavior when relevant
 
 ---
 
-## If someone asks for deeper proof
+## 4. E2E QA verdict
 
-Expand the proof group and walk in this order:
+Call the demo **interactive V7 E2E-QA ready** only if all of the following are true:
 
-1. ranking explanation
-2. conservation check
-3. public receipts
-4. receipt schemas
-5. policy release / attestation
-6. proof log timeline
+- [ ] page loads cleanly
+- [ ] reset works
+- [ ] make-ENGI-branch gold path works end-to-end
+- [ ] V7 profile/derivation/rights/telemetry surfaces are visible
+- [ ] settlement invariants hold
+- [ ] no stale query-first story is required to understand the demo
+- [ ] docs/checklist match the actual UI/runtime behavior
 
 ---
 
-## If something looks wrong
+## 5. If something is wrong
 
-### If state is dirty
+### If state looks dirty
 - [ ] Click **Reset demo**
-- [ ] Reconfirm 0 receipts
+- [ ] Re-run the gold path
 
-### If proof group is distracting
-- [ ] Switch to `light` mode
-
-### If the benchmark panel is not the last visual beat
-- [ ] Scroll back to `3. Utility proof`
-- [ ] re-land the value claim before closing
+### If the UI seems stale
+- [ ] Hard refresh browser
+- [ ] Verify `data/state.json` was regenerated from the current V7 model if needed
 
 ### If the server is not responding
-Run from repo root:
 
 ```bash
-cd /Users/garrettmaring/.openclaw/workspace/engi-demo
-node server.js
+cd /Users/garrettmaring/Developer/ENGI/engi-demo
+npm start
 ```
 
-Then reload:
-
-`http://127.0.0.1:4318`
+### If invariants fail
+- [ ] stop the demo
+- [ ] run `npm test`
+- [ ] do not present the prototype as QA-ready until the failure is understood
 
 ---
 
 ## Final reminder
 
-Do not end on proof mechanics.
-
-End on this:
+Do not present the demo as:
+- a live GitHub-integrated production system
+- a real embedding/LLM evaluator deployment
+- a real policy publication / authz enforcement plane
 
-- the buyer had a real production auth incident
-- the system assembled the right licensed bundle
-- the read was provable
-- the read improved the buyer's remediation system
+Present it as:
+- a **Spec V7 Profile A deterministic prototype**
+- with explicit **Profile B production-boundary intent**
+- and with strong inspectability across ranking, verification, artifacts, telemetry, and settlement

engi-demo/CHECKLIST.md

@@ -1,10 +1,10 @@
-# ENGI Demo — Spec V6 Deterministic Local Prototype
+# ENGI Demo — Spec V7 Deterministic Local Prototype
 
-This demo now tracks the Spec V6 core flow much more closely:
+This demo now tracks the Spec V7 core flow much more closely:
 
 1. parse mocked GitHub Actions evidence through a declared benchmark parser contract and build a `GitHubNeedDescriptor`,
 2. model hybrid candidate recall across task, failure-mode, technical-context, lexical, symbol, path, config, and artifact-kind channels,
-3. rank deposited candidate assets with explicit Spec V6 subscores, explainability traces, and penalties,
+3. rank deposited candidate assets with explicit Spec V7 subscores, explainability traces, and penalties,
 4. apply separate verification determinisms, issuer policy, and downstream use-tier propagation,
 5. assemble a locked `AssetPack` plus selected source-material manifest,
 6. stage a private remediation branch artifact set with deliverable metadata, authorization decisions, and sensitive-data-flow records,
@@ -13,18 +13,22 @@ This demo now tracks the Spec V6 core flow much more closely:
 
 ## What this prototype is
 
-A deterministic, local, no-dependency Node app that models the V6 architecture and naming faithfully enough to demo:
+A deterministic, local, no-dependency Node app that models the V7 Profile A architecture faithfully enough to demo while explicitly labeling Profile B production intent:
 
 - need measurement from GitHub-bound benchmark evidence,
 - parser fail-closed behavior and canonical run-evidence normalization,
+- canonical need-descriptor derivation closure with explicit field derivations,
+- content-unit semantics with explicit embedding/vector hand-off interfaces,
 - need-measurement provenance plus inference proof surfaces,
+- clean separation between static deterministic analysis and inferred evaluator surfaces,
 - hybrid candidate recall and fusion provenance,
 - ranking vs verification separation,
 - issuer policy / sufficiency / use-tier propagation,
-- branch materialization rules by use tier,
-- authorization, identity binding, confidentiality, retention, and revocation modeling,
+- branch materialization rules by use tier and branch mode,
+- authorization, identity binding, confidentiality, retention, revocation, and bounded-public-proof modeling,
+- structured pipeline telemetry and unit catalogs for live demo inspection,
 - private remediation branch artifacts under `.engi/`,
-- exact fixed-point settlement with asset-scoped pending claims.
+- exact fixed-point settlement with asset-scoped pending claims and asset-pack-lock binding.
 
 ## What it still is not
 
@@ -39,6 +43,18 @@ This is still a local prototype. It does **not**:
 
 All of those surfaces are represented deterministically in local state.
 
+## Conformance profiles
+
+- `Profile A — local deterministic prototype`: what this repo implements and demos now.
+- `Profile B — production-boundary intent`: live GitHub binding, real branch/privacy/authz enforcement, real embeddings/evaluators/prompts, and external integrations still outside the local prototype boundary.
+
+## Advanced V7 interface notes
+
+- Content units now carry explicit semantic/embedding contracts so real embedding providers can replace deterministic stand-ins without changing downstream schemas.
+- Measurement provenance distinguishes static deterministic analysis from inferred evaluator surfaces.
+- Structured telemetry artifacts (`.engi/unit-catalog.json`, `.engi/pipeline-telemetry.json`) explain the live pipeline during demos.
+- Proof bundles now include prompt/evaluator implementation surfaces alongside the existing settlement / policy / identity proofs.
+
 ## Run
 
 ```bash
@@ -79,12 +95,15 @@ npm test
 - `.engi/authorization-decisions.json`
 - `.engi/sensitive-data-flow.json`
 - `.engi/policy-release.json`
+- `.engi/unit-catalog.json`
+- `.engi/pipeline-telemetry.json`
 - `.engi/deliverables.json`
 - `.engi/source-material/*`
 - `ENGI_NEED.md`
 
 ## Repo docs added for this alignment pass
 
 - `ARCHITECTURE_MAP.md` — tight map of the original/current demo architecture baseline
-- `SPEC_V6_GAP_ANALYSIS.md` — gap analysis and implementation guide used for the refactor
+- `SPEC_V6_GAP_ANALYSIS.md` — earlier gap analysis and implementation guide used for the V6 refactor
 - `SPEC_V6_COVERAGE_MATRIX.md` — durable section-by-section coverage audit against the V6 spec
+- `SPEC_V7_COVERAGE_MATRIX.md` — section-by-section coverage audit against the V7 spec, including V6→V7 deltas and Profile A/Profile B splits