Skip to content

Commit 06c9c1a

Browse files
V40 Gate 10: Prompt Benchmark Smoke And V41 Readiness
Adds source-safe PromptPart and composed Prompt benchmark smoke automation, generated protocol artifact, workflow wiring, and Gate 10 checks. Updates the V40 spec family, roadmap, and package documentation to hand focused prompt-program evolution to V41.
1 parent 5266be5 commit 06c9c1a

18 files changed

Lines changed: 1905 additions & 15 deletions

.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json

Lines changed: 672 additions & 0 deletions
Large diffs are not rendered by default.

.github/workflows/bitcode-canon-quality.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -247,6 +247,9 @@ jobs:
247247
if [ -f scripts/check-v40-gate9-local-staging-rehearsal-automation.mjs ]; then
248248
node scripts/check-v40-gate9-local-staging-rehearsal-automation.mjs --skip-branch-check --skip-package-tests
249249
fi
250+
if [ -f scripts/check-v40-gate10-prompt-benchmark-smoke-v41-readiness.mjs ]; then
251+
node scripts/check-v40-gate10-prompt-benchmark-smoke-v41-readiness.mjs --skip-branch-check --skip-package-tests
252+
fi
250253
fi
251254
else
252255
echo "Unexpected BITCODE_SPEC.txt pointer: $POINTER" >&2

.github/workflows/bitcode-gate-quality.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -376,6 +376,9 @@ jobs:
376376
if [ -f scripts/check-v40-gate9-local-staging-rehearsal-automation.mjs ]; then
377377
node scripts/check-v40-gate9-local-staging-rehearsal-automation.mjs --skip-branch-check --skip-package-tests
378378
fi
379+
if [ -f scripts/check-v40-gate10-prompt-benchmark-smoke-v41-readiness.mjs ]; then
380+
node scripts/check-v40-gate10-prompt-benchmark-smoke-v41-readiness.mjs --skip-branch-check --skip-package-tests
381+
fi
379382
fi
380383
else
381384
echo "Unexpected BITCODE_SPEC.txt pointer: $POINTER" >&2
@@ -508,6 +511,9 @@ jobs:
508511
if [ -f packages/protocol/test/v40-local-staging-rehearsal-automation.test.js ]; then
509512
pnpm --dir packages/protocol exec node --test --test-force-exit test/v40-local-staging-rehearsal-automation.test.js
510513
fi
514+
if [ -f packages/protocol/test/v40-prompt-benchmark-smoke-v41-readiness.test.js ]; then
515+
pnpm --dir packages/protocol exec node --test --test-force-exit test/v40-prompt-benchmark-smoke-v41-readiness.test.js
516+
fi
511517
pnpm --filter @bitcode/protocol test
512518
513519
- name: Test staged Reading route and Terminal harness contracts

BITCODE_SPEC_V40.md

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
- Prior canonical anchor: `BITCODE_SPEC_V39.md`
1010
- Prior generated proof appendix: `BITCODE_SPEC_V39_PROVEN.md`
1111
- Active canonical pointer during draft opening: `BITCODE_SPEC.txt` -> `V39`
12-
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, plus later V40 gate artifacts for synchronization, rehearsal, prompt benchmark, and promotion readiness coverage
12+
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, `.bitcode/v40-ledger-storage-sync.json`, `.bitcode/v40-local-staging-rehearsal-automation.json`, `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, and later V40 promotion readiness coverage
1313
- Source parity state: V40 opens testing-system specification over active V39; package, API, pipeline, conversation, Terminal, Auxillaries, Exchange, interface, ledger/database/storage, and demonstration tests remain to be hardened gate by gate
1414
- Notes companion: `BITCODE_SPEC_V40_NOTES.md`
1515
- Delta companion: `BITCODE_SPEC_V40_DELTA.md`
@@ -160,6 +160,14 @@ The operator command `rehearse:v40-local-staging` builds local or staging-testne
160160
Staging-testnet automation binds the Supabase project `tkpyosihuouusyaxtbau`, the REST host `https://tkpyosihuouusyaxtbau.supabase.co/rest/v1/`, real inference, database event streaming, and structured readback while serializing no secret values, protected source, raw prompts, provider responses, unpaid AssetPack source, wallet private material, private settlement payloads, or live log payloads.
161161
Gate 9 permits no missing critical rehearsal automation row and keeps value-bearing mainnet admission blocked.
162162

163+
## V40 Gate 10 Prompt Benchmark Smoke And V41 Readiness
164+
165+
Gate 10 closes source-safe prompt benchmark smoke coverage before V41 begins focused prompt-program rewriting.
166+
The package-owned `V40PromptBenchmarkSmokeV41Readiness` emits `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` with ten source-safe rows covering the prompt benchmark report command, deterministic PromptPart smoke execution, deterministic composed Prompt smoke execution, V38 benchmark inventory binding, Reading prompt fixtures, Conversation and tool prompt fixtures, the no-prompt-rewrite boundary, the V41 prompt-program worklist, workflow wiring, and proof-system wiring.
167+
The operator command `prompt-benchmark:smoke` runs a local deterministic mock benchmark receipt and confirms the package benchmark report command remains runnable.
168+
Gate 10 serializes no raw prompt text, protected PromptPart source, raw provider response, credential, wallet private material, private settlement payload, or unpaid AssetPack source.
169+
Gate 10 intentionally proves benchmark infrastructure readiness only; V41 owns every raw PromptPart and composed Prompt audit, semantic repartition, retitle, rewrite, catalogue, interpolation binding review, and callsite-quality benchmark.
170+
163171
## V40 canonical subsystem surfaces
164172

165173
### Depositing and asset supply
@@ -249,7 +257,7 @@ Current accepted boundaries: proof artifacts may summarize sensitive facts but c
249257
| proofFamily | proofArtifactPath | memberIds | theoremIds | replayStepIds | witnessArtifactPaths | Current source basis |
250258
| --- | --- | --- | --- | --- | --- | --- |
251259
| Inference-synthesis | `.bitcode/v40-reading-pipeline-integration-coverage.json` | pipeline, agent, step | inference-stack-covered | run-reading-pipeline-tests | prompt receipts, generation receipts | packages/pipelines, packages/agent-generics |
252-
| Prompt-completeness | `.bitcode/v40-prompt-benchmark-smoke.json` | PromptParts, Prompts | prompt-benchmark-runnable | run-prompt-benchmark-smoke | benchmark receipts | packages/prompts |
260+
| Prompt-completeness | `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` | PromptParts, Prompts | prompt-benchmark-runnable | run-prompt-benchmark-smoke | benchmark receipts | packages/prompts |
253261
| Static-code-analysis | `.bitcode/v40-unit-coverage-inventory.json` | source packages | source-units-covered | run-unit-suites | coverage maps | packages, uapi |
254262
| Verification-decisions | `.bitcode/v40-api-integration-contracts.json` | routes, contracts | route-contracts-covered | run-api-integration | API receipts | uapi, packages/api |
255263
| Selection-and-materialization | `.bitcode/v40-browser-e2e-visual-proof.json` | browser flows | ux-state-covered | run-browser-visual | screenshot baselines | uapi tests |
@@ -276,7 +284,7 @@ fail-closed conditions: missing prompt, invalid typed output, or unbound tool re
276284

277285
### Prompt-completeness
278286

279-
proofArtifactPath: `.bitcode/v40-prompt-benchmark-smoke.json`
287+
proofArtifactPath: `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`
280288
members: PromptParts, Prompts, templates, interpolation bindings
281289
theoremIds: prompt-benchmark-runnable
282290
replayStepIds: run-prompt-benchmark-smoke
@@ -286,8 +294,8 @@ current member verdict shape: pass, fail, missing fixture, or deferred-to-V41
286294
current theorem-by-theorem closure reading: V40 proves benchmark execution, V41 improves prompt content
287295
current theorem-to-replay grouping: PromptPart and composed Prompt suites
288296
minimum artifact/replay binding set: prompt id, PromptPart id, fixture id, result root
289-
current proof-object fields: promptId, promptPartIds, fixtureRoot, qualityVerdict
290-
generated-artifact and test bindings: prompt benchmark smoke artifact
297+
current proof-object fields: promptId, promptPartIds, fixtureRoot, qualityVerdict, v41WorkItemIds
298+
generated-artifact and test bindings: `V40PromptBenchmarkSmokeV41Readiness`, prompt benchmark smoke artifact, and V41 prompt-program handoff
291299
fail-closed conditions: missing prompt identity or unbound interpolation
292300

293301
### Static-code-analysis
@@ -426,12 +434,13 @@ V40 inherits operator-quality artifact expectations for readable proof and repai
426434
| `.bitcode/v40-browser-e2e-visual-proof.json` | protocol canonical package | Gate 7 source-safe browser, visual, accessibility, and responsive proof |
427435
| `.bitcode/v40-ledger-storage-sync.json` | protocol canonical package | Gate 8 source-safe ledger, database, storage, wallet, and delivery synchronization proof |
428436
| `.bitcode/v40-local-staging-rehearsal-automation.json` | protocol canonical package | Gate 9 source-safe local and staging-testnet rehearsal automation proof |
437+
| `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` | protocol canonical package | Gate 10 source-safe PromptPart and Prompt benchmark smoke plus V41 prompt-program readiness proof |
429438
| `.bitcode/v40-promotion-readiness-report.json` | future V40 promotion gate | promotion readiness |
430439

431440
### V40 specifying generated artifacts
432441

433442
V40 generated artifacts begin with `.bitcode/v40-spec-family-report.json` and `.bitcode/v40-canonical-input-report.json`.
434-
Closed gates now add unit, API, pipeline, Conversation/Terminal, browser, synchronization, and local/staging rehearsal artifacts; later gates add prompt benchmark smoke and promotion-readiness artifacts.
443+
Closed gates now add unit, API, pipeline, Conversation/Terminal, browser, synchronization, local/staging rehearsal, and prompt benchmark smoke artifacts; later gates add promotion-readiness artifacts.
435444

436445
### Shared generated-artifact fields
437446

@@ -457,7 +466,7 @@ Missing generated artifacts, stale reports, malformed JSON, invalid screenshots,
457466

458467
## V40 validation canon
459468

460-
Validation canon includes `pnpm run check:v40-gate1`, `pnpm run check:v40-gate2`, `pnpm run check:v40-gate3`, `pnpm run check:v40-gate4`, `pnpm run check:v40-gate5`, `pnpm run check:v40-gate6`, `pnpm run check:v40-gate7`, `pnpm run check:v40-gate8`, `pnpm run check:v40-gate9`, `pnpm run check:v40-browser-e2e-visual-proof`, `pnpm run rehearse:v40-local-staging --lane local --dry-run --json`, `pnpm run rehearse:v40-local-staging --lane staging-testnet --dry-run --json`, `node scripts/check-bitcode-spec-family.mjs --version V40 --mode draft --current-target V39`, `node scripts/check-bitcode-canon-posture-drift.mjs --active-canon V39 --draft-target V40`, package typechecks, protocol tests, route/API tests, browser E2E suites, visual/screenshot proof suites, integration suites, unit suites, prompt benchmark smoke tests, local/staging rehearsal commands, and diff hygiene.
469+
Validation canon includes `pnpm run check:v40-gate1`, `pnpm run check:v40-gate2`, `pnpm run check:v40-gate3`, `pnpm run check:v40-gate4`, `pnpm run check:v40-gate5`, `pnpm run check:v40-gate6`, `pnpm run check:v40-gate7`, `pnpm run check:v40-gate8`, `pnpm run check:v40-gate9`, `pnpm run check:v40-gate10`, `pnpm run check:v40-browser-e2e-visual-proof`, `pnpm run prompt-benchmark:smoke`, `pnpm run generate:v40-prompt-benchmark-smoke-v41-readiness`, `pnpm run check:v40-prompt-benchmark-smoke-v41-readiness`, `pnpm run rehearse:v40-local-staging --lane local --dry-run --json`, `pnpm run rehearse:v40-local-staging --lane staging-testnet --dry-run --json`, `node scripts/check-bitcode-spec-family.mjs --version V40 --mode draft --current-target V39`, `node scripts/check-bitcode-canon-posture-drift.mjs --active-canon V39 --draft-target V40`, package typechecks, protocol tests, route/API tests, browser E2E suites, visual/screenshot proof suites, integration suites, unit suites, prompt benchmark smoke tests, local/staging rehearsal commands, and diff hygiene.
461470

462471
## V40 promotion canon
463472

BITCODE_SPEC_V40_DELTA.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
- Prior canonical anchor: `BITCODE_SPEC_V39.md`
99
- Prior generated proof appendix: `BITCODE_SPEC_V39_PROVEN.md`
1010
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, and later V40 testing artifacts
11-
- Source parity state: V40 testing surfaces are scoped but not yet closed
11+
- Source parity state: V40 testing surfaces are closing progressively through package-backed proof artifacts
1212

1313
## Why V40 exists
1414

@@ -73,6 +73,8 @@ Gate 9 closes with package-backed `V40LocalStagingRehearsalAutomation`, determin
7373
### Gate 10: Prompt Benchmark Smoke And V41 Readiness
7474

7575
Make prompt and PromptPart benchmark commands runnable and reportable without rewriting prompt content, then hand focused prompt evolution to V41.
76+
Gate 10 closes with package-backed `V40PromptBenchmarkSmokeV41Readiness`, deterministic `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, protocol exports, source-safe PromptPart and composed Prompt smoke receipts, package benchmark report command binding, V38 prompt benchmark inventory binding, workflow wiring, and `check:v40-gate10`.
77+
Gate 10 does not rewrite PromptPart or Prompt content; it proves benchmark infrastructure readiness and hands every semantic prompt-program improvement to V41.
7678

7779
### Gate 11: V40 Promotion Readiness
7880

BITCODE_SPEC_V40_NOTES.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
- Current canonical/latest target: `V39`
88
- Prior canonical anchor: `BITCODE_SPEC_V39.md`
99
- Prior generated proof appendix: `BITCODE_SPEC_V39_PROVEN.md`
10-
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, and later V40 testing artifacts
10+
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, `.bitcode/v40-ledger-storage-sync.json`, `.bitcode/v40-local-staging-rehearsal-automation.json`, `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, and later V40 promotion readiness artifacts
1111
- Source parity state: V40 testing gates are closing progressively through generated artifacts
1212

1313
## Notes companion rule
@@ -122,3 +122,10 @@ Its generated `.bitcode/v40-local-staging-rehearsal-automation.json` artifact bi
122122
The operator command `rehearse:v40-local-staging` emits receipts from environment presence only; secret values, protected source, raw prompts, raw provider responses, unpaid AssetPack source, wallet private material, private settlement payloads, and live log payloads are absent from the generated artifact and receipts.
123123
Live execution remains dry-run by default and requires `BITCODE_V40_REHEARSAL_EXECUTE=1` before delegating to the Vercel Sandbox AssetPack harness.
124124
Staging-testnet receipts bind Supabase project `tkpyosihuouusyaxtbau`, REST host `https://tkpyosihuouusyaxtbau.supabase.co/rest/v1/`, real inference, and database event streaming/readback.
125+
126+
## Gate 10 implementation notes
127+
128+
Gate 10 is closed by `V40PromptBenchmarkSmokeV41Readiness`.
129+
Its generated `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` artifact binds prompt benchmark smoke coverage to ten source-safe rows: benchmark report command execution, deterministic PromptPart smoke execution, deterministic composed Prompt smoke execution, V38 benchmark inventory binding, Reading prompt fixtures, Conversation and tool prompt fixtures, source-safe no-rewrite boundary, V41 prompt-program worklist, workflow wiring, and proof-system wiring.
130+
The `prompt-benchmark:smoke` command uses a local source-safe mock provider and records benchmark scores, fixture roots, and package benchmark report counts without serializing raw prompt text, protected PromptPart source, raw provider responses, credentials, wallet private material, private settlement payloads, or unpaid AssetPack source.
131+
Gate 10 exists to make prompt and PromptPart benchmarking runnable and auditable before V41; every prompt semantic repartition, retitle, rewrite, catalogue improvement, interpolation binding review, and callsite-quality benchmark remains V41 work.

BITCODE_SPEC_V40_PARITY_MATRIX.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@
77
- Current canonical/latest target: `V39`
88
- Prior canonical anchor: `BITCODE_SPEC_V39.md`
99
- Prior generated proof appendix: `BITCODE_SPEC_V39_PROVEN.md`
10-
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, and later V40 testing artifacts
11-
- Source parity state: V40 testing parity is drafted
10+
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, `.bitcode/v40-ledger-storage-sync.json`, `.bitcode/v40-local-staging-rehearsal-automation.json`, `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, and later V40 promotion readiness artifacts
11+
- Source parity state: V40 testing parity is closing progressively through generated artifacts
1212

1313
## Purpose
1414

@@ -36,7 +36,7 @@ This matrix records the V40 testing surfaces that must become promotion-grade be
3636
| Pipeline integration | Primitive and real Reading pipeline implementations are tested | `.bitcode/v40-reading-pipeline-integration-coverage.json` | implemented |
3737
| Conversation/Terminal integration | Conversation handoff, stream logs, Terminal Reading state, and authority boundaries are tested source-safely | `.bitcode/v40-conversation-terminal-integration.json` | implemented |
3838
| Unit coverage | Packages, primitives, isolated implementations, and real implementations have unit coverage | `.bitcode/v40-unit-coverage-inventory.json` | implemented |
39-
| Prompt benchmark smoke | Prompt and PromptPart benchmarks run before V41 | future V40 artifacts | not yet implemented |
39+
| Prompt benchmark smoke | Prompt and PromptPart benchmarks run before V41 | `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` and `V40PromptBenchmarkSmokeV41Readiness` | implemented |
4040

4141
## V40 implementation checklist
4242

@@ -51,7 +51,7 @@ This matrix records the V40 testing surfaces that must become promotion-grade be
5151
| Gate 7 | Browser/visual/accessibility/responsive artifact | implemented |
5252
| Gate 8 | Ledger/database/storage/wallet/delivery sync artifact | implemented |
5353
| Gate 9 | Local/staging rehearsal artifact | implemented |
54-
| Gate 10 | Prompt benchmark smoke and V41 readiness artifact | not yet implemented |
54+
| Gate 10 | Prompt benchmark smoke and V41 readiness artifact | implemented |
5555
| Gate 11 | Promotion readiness artifact and workflow | not yet implemented |
5656

5757
Gate 2 implementation evidence: package-backed `V40TestInventoryCoverageMatrix` emits `.bitcode/v40-test-inventory-coverage-matrix.json` and is wired through `check:v40-gate2`.
@@ -62,6 +62,7 @@ Gate 6 implementation evidence: package-backed `V40ConversationTerminalIntegrati
6262
Gate 7 implementation evidence: package-backed `V40BrowserE2eVisualProof` emits `.bitcode/v40-browser-e2e-visual-proof.json` and is wired through `check:v40-gate7`.
6363
Gate 8 implementation evidence: package-backed `V40LedgerStorageSync` emits `.bitcode/v40-ledger-storage-sync.json` and is wired through `check:v40-gate8`.
6464
Gate 9 implementation evidence: package-backed `V40LocalStagingRehearsalAutomation` emits `.bitcode/v40-local-staging-rehearsal-automation.json` and is wired through `check:v40-gate9`.
65+
Gate 10 implementation evidence: package-backed `V40PromptBenchmarkSmokeV41Readiness` emits `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, runs source-safe PromptPart and composed Prompt smoke receipts through `prompt-benchmark:smoke`, binds V38 benchmark inventory evidence, and is wired through `check:v40-gate10`.
6566

6667
## V40 accepted boundaries
6768

0 commit comments

Comments
 (0)