Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
672 changes: 672 additions & 0 deletions .bitcode/v40-prompt-benchmark-smoke-v41-readiness.json

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions .github/workflows/bitcode-canon-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,9 @@ jobs:
if [ -f scripts/check-v40-gate9-local-staging-rehearsal-automation.mjs ]; then
node scripts/check-v40-gate9-local-staging-rehearsal-automation.mjs --skip-branch-check --skip-package-tests
fi
if [ -f scripts/check-v40-gate10-prompt-benchmark-smoke-v41-readiness.mjs ]; then
node scripts/check-v40-gate10-prompt-benchmark-smoke-v41-readiness.mjs --skip-branch-check --skip-package-tests
fi
fi
else
echo "Unexpected BITCODE_SPEC.txt pointer: $POINTER" >&2
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/bitcode-gate-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,9 @@ jobs:
if [ -f scripts/check-v40-gate9-local-staging-rehearsal-automation.mjs ]; then
node scripts/check-v40-gate9-local-staging-rehearsal-automation.mjs --skip-branch-check --skip-package-tests
fi
if [ -f scripts/check-v40-gate10-prompt-benchmark-smoke-v41-readiness.mjs ]; then
node scripts/check-v40-gate10-prompt-benchmark-smoke-v41-readiness.mjs --skip-branch-check --skip-package-tests
fi
fi
else
echo "Unexpected BITCODE_SPEC.txt pointer: $POINTER" >&2
Expand Down Expand Up @@ -508,6 +511,9 @@ jobs:
if [ -f packages/protocol/test/v40-local-staging-rehearsal-automation.test.js ]; then
pnpm --dir packages/protocol exec node --test --test-force-exit test/v40-local-staging-rehearsal-automation.test.js
fi
if [ -f packages/protocol/test/v40-prompt-benchmark-smoke-v41-readiness.test.js ]; then
pnpm --dir packages/protocol exec node --test --test-force-exit test/v40-prompt-benchmark-smoke-v41-readiness.test.js
fi
pnpm --filter @bitcode/protocol test

- name: Test staged Reading route and Terminal harness contracts
Expand Down
23 changes: 16 additions & 7 deletions BITCODE_SPEC_V40.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
- Prior canonical anchor: `BITCODE_SPEC_V39.md`
- Prior generated proof appendix: `BITCODE_SPEC_V39_PROVEN.md`
- Active canonical pointer during draft opening: `BITCODE_SPEC.txt` -> `V39`
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, plus later V40 gate artifacts for synchronization, rehearsal, prompt benchmark, and promotion readiness coverage
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, `.bitcode/v40-ledger-storage-sync.json`, `.bitcode/v40-local-staging-rehearsal-automation.json`, `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, and later V40 promotion readiness coverage
- Source parity state: V40 opens testing-system specification over active V39; package, API, pipeline, conversation, Terminal, Auxillaries, Exchange, interface, ledger/database/storage, and demonstration tests remain to be hardened gate by gate
- Notes companion: `BITCODE_SPEC_V40_NOTES.md`
- Delta companion: `BITCODE_SPEC_V40_DELTA.md`
Expand Down Expand Up @@ -160,6 +160,14 @@ The operator command `rehearse:v40-local-staging` builds local or staging-testne
Staging-testnet automation binds the Supabase project `tkpyosihuouusyaxtbau`, the REST host `https://tkpyosihuouusyaxtbau.supabase.co/rest/v1/`, real inference, database event streaming, and structured readback while serializing no secret values, protected source, raw prompts, provider responses, unpaid AssetPack source, wallet private material, private settlement payloads, or live log payloads.
Gate 9 permits no missing critical rehearsal automation row and keeps value-bearing mainnet admission blocked.

## V40 Gate 10 Prompt Benchmark Smoke And V41 Readiness

Gate 10 closes source-safe prompt benchmark smoke coverage before V41 begins focused prompt-program rewriting.
The package-owned `V40PromptBenchmarkSmokeV41Readiness` emits `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` with ten source-safe rows covering the prompt benchmark report command, deterministic PromptPart smoke execution, deterministic composed Prompt smoke execution, V38 benchmark inventory binding, Reading prompt fixtures, Conversation and tool prompt fixtures, the no-prompt-rewrite boundary, the V41 prompt-program worklist, workflow wiring, and proof-system wiring.
The operator command `prompt-benchmark:smoke` runs a local deterministic mock benchmark receipt and confirms the package benchmark report command remains runnable.
Gate 10 serializes no raw prompt text, protected PromptPart source, raw provider response, credential, wallet private material, private settlement payload, or unpaid AssetPack source.
Gate 10 intentionally proves benchmark infrastructure readiness only; V41 owns every raw PromptPart and composed Prompt audit, semantic repartition, retitle, rewrite, catalogue, interpolation binding review, and callsite-quality benchmark.

## V40 canonical subsystem surfaces

### Depositing and asset supply
Expand Down Expand Up @@ -249,7 +257,7 @@ Current accepted boundaries: proof artifacts may summarize sensitive facts but c
| proofFamily | proofArtifactPath | memberIds | theoremIds | replayStepIds | witnessArtifactPaths | Current source basis |
| --- | --- | --- | --- | --- | --- | --- |
| Inference-synthesis | `.bitcode/v40-reading-pipeline-integration-coverage.json` | pipeline, agent, step | inference-stack-covered | run-reading-pipeline-tests | prompt receipts, generation receipts | packages/pipelines, packages/agent-generics |
| Prompt-completeness | `.bitcode/v40-prompt-benchmark-smoke.json` | PromptParts, Prompts | prompt-benchmark-runnable | run-prompt-benchmark-smoke | benchmark receipts | packages/prompts |
| Prompt-completeness | `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` | PromptParts, Prompts | prompt-benchmark-runnable | run-prompt-benchmark-smoke | benchmark receipts | packages/prompts |
| Static-code-analysis | `.bitcode/v40-unit-coverage-inventory.json` | source packages | source-units-covered | run-unit-suites | coverage maps | packages, uapi |
| Verification-decisions | `.bitcode/v40-api-integration-contracts.json` | routes, contracts | route-contracts-covered | run-api-integration | API receipts | uapi, packages/api |
| Selection-and-materialization | `.bitcode/v40-browser-e2e-visual-proof.json` | browser flows | ux-state-covered | run-browser-visual | screenshot baselines | uapi tests |
Expand All @@ -276,7 +284,7 @@ fail-closed conditions: missing prompt, invalid typed output, or unbound tool re

### Prompt-completeness

proofArtifactPath: `.bitcode/v40-prompt-benchmark-smoke.json`
proofArtifactPath: `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`
members: PromptParts, Prompts, templates, interpolation bindings
theoremIds: prompt-benchmark-runnable
replayStepIds: run-prompt-benchmark-smoke
Expand All @@ -286,8 +294,8 @@ current member verdict shape: pass, fail, missing fixture, or deferred-to-V41
current theorem-by-theorem closure reading: V40 proves benchmark execution, V41 improves prompt content
current theorem-to-replay grouping: PromptPart and composed Prompt suites
minimum artifact/replay binding set: prompt id, PromptPart id, fixture id, result root
current proof-object fields: promptId, promptPartIds, fixtureRoot, qualityVerdict
generated-artifact and test bindings: prompt benchmark smoke artifact
current proof-object fields: promptId, promptPartIds, fixtureRoot, qualityVerdict, v41WorkItemIds
generated-artifact and test bindings: `V40PromptBenchmarkSmokeV41Readiness`, prompt benchmark smoke artifact, and V41 prompt-program handoff
fail-closed conditions: missing prompt identity or unbound interpolation

### Static-code-analysis
Expand Down Expand Up @@ -426,12 +434,13 @@ V40 inherits operator-quality artifact expectations for readable proof and repai
| `.bitcode/v40-browser-e2e-visual-proof.json` | protocol canonical package | Gate 7 source-safe browser, visual, accessibility, and responsive proof |
| `.bitcode/v40-ledger-storage-sync.json` | protocol canonical package | Gate 8 source-safe ledger, database, storage, wallet, and delivery synchronization proof |
| `.bitcode/v40-local-staging-rehearsal-automation.json` | protocol canonical package | Gate 9 source-safe local and staging-testnet rehearsal automation proof |
| `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` | protocol canonical package | Gate 10 source-safe PromptPart and Prompt benchmark smoke plus V41 prompt-program readiness proof |
| `.bitcode/v40-promotion-readiness-report.json` | future V40 promotion gate | promotion readiness |

### V40 specifying generated artifacts

V40 generated artifacts begin with `.bitcode/v40-spec-family-report.json` and `.bitcode/v40-canonical-input-report.json`.
Closed gates now add unit, API, pipeline, Conversation/Terminal, browser, synchronization, and local/staging rehearsal artifacts; later gates add prompt benchmark smoke and promotion-readiness artifacts.
Closed gates now add unit, API, pipeline, Conversation/Terminal, browser, synchronization, local/staging rehearsal, and prompt benchmark smoke artifacts; later gates add promotion-readiness artifacts.

### Shared generated-artifact fields

Expand All @@ -457,7 +466,7 @@ Missing generated artifacts, stale reports, malformed JSON, invalid screenshots,

## V40 validation canon

Validation canon includes `pnpm run check:v40-gate1`, `pnpm run check:v40-gate2`, `pnpm run check:v40-gate3`, `pnpm run check:v40-gate4`, `pnpm run check:v40-gate5`, `pnpm run check:v40-gate6`, `pnpm run check:v40-gate7`, `pnpm run check:v40-gate8`, `pnpm run check:v40-gate9`, `pnpm run check:v40-browser-e2e-visual-proof`, `pnpm run rehearse:v40-local-staging --lane local --dry-run --json`, `pnpm run rehearse:v40-local-staging --lane staging-testnet --dry-run --json`, `node scripts/check-bitcode-spec-family.mjs --version V40 --mode draft --current-target V39`, `node scripts/check-bitcode-canon-posture-drift.mjs --active-canon V39 --draft-target V40`, package typechecks, protocol tests, route/API tests, browser E2E suites, visual/screenshot proof suites, integration suites, unit suites, prompt benchmark smoke tests, local/staging rehearsal commands, and diff hygiene.
Validation canon includes `pnpm run check:v40-gate1`, `pnpm run check:v40-gate2`, `pnpm run check:v40-gate3`, `pnpm run check:v40-gate4`, `pnpm run check:v40-gate5`, `pnpm run check:v40-gate6`, `pnpm run check:v40-gate7`, `pnpm run check:v40-gate8`, `pnpm run check:v40-gate9`, `pnpm run check:v40-gate10`, `pnpm run check:v40-browser-e2e-visual-proof`, `pnpm run prompt-benchmark:smoke`, `pnpm run generate:v40-prompt-benchmark-smoke-v41-readiness`, `pnpm run check:v40-prompt-benchmark-smoke-v41-readiness`, `pnpm run rehearse:v40-local-staging --lane local --dry-run --json`, `pnpm run rehearse:v40-local-staging --lane staging-testnet --dry-run --json`, `node scripts/check-bitcode-spec-family.mjs --version V40 --mode draft --current-target V39`, `node scripts/check-bitcode-canon-posture-drift.mjs --active-canon V39 --draft-target V40`, package typechecks, protocol tests, route/API tests, browser E2E suites, visual/screenshot proof suites, integration suites, unit suites, prompt benchmark smoke tests, local/staging rehearsal commands, and diff hygiene.

## V40 promotion canon

Expand Down
4 changes: 3 additions & 1 deletion BITCODE_SPEC_V40_DELTA.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
- Prior canonical anchor: `BITCODE_SPEC_V39.md`
- Prior generated proof appendix: `BITCODE_SPEC_V39_PROVEN.md`
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, and later V40 testing artifacts
- Source parity state: V40 testing surfaces are scoped but not yet closed
- Source parity state: V40 testing surfaces are closing progressively through package-backed proof artifacts

## Why V40 exists

Expand Down Expand Up @@ -73,6 +73,8 @@ Gate 9 closes with package-backed `V40LocalStagingRehearsalAutomation`, determin
### Gate 10: Prompt Benchmark Smoke And V41 Readiness

Make prompt and PromptPart benchmark commands runnable and reportable without rewriting prompt content, then hand focused prompt evolution to V41.
Gate 10 closes with package-backed `V40PromptBenchmarkSmokeV41Readiness`, deterministic `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, protocol exports, source-safe PromptPart and composed Prompt smoke receipts, package benchmark report command binding, V38 prompt benchmark inventory binding, workflow wiring, and `check:v40-gate10`.
Gate 10 does not rewrite PromptPart or Prompt content; it proves benchmark infrastructure readiness and hands every semantic prompt-program improvement to V41.

### Gate 11: V40 Promotion Readiness

Expand Down
9 changes: 8 additions & 1 deletion BITCODE_SPEC_V40_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
- Current canonical/latest target: `V39`
- Prior canonical anchor: `BITCODE_SPEC_V39.md`
- Prior generated proof appendix: `BITCODE_SPEC_V39_PROVEN.md`
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, and later V40 testing artifacts
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, `.bitcode/v40-ledger-storage-sync.json`, `.bitcode/v40-local-staging-rehearsal-automation.json`, `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, and later V40 promotion readiness artifacts
- Source parity state: V40 testing gates are closing progressively through generated artifacts

## Notes companion rule
Expand Down Expand Up @@ -122,3 +122,10 @@ Its generated `.bitcode/v40-local-staging-rehearsal-automation.json` artifact bi
The operator command `rehearse:v40-local-staging` emits receipts from environment presence only; secret values, protected source, raw prompts, raw provider responses, unpaid AssetPack source, wallet private material, private settlement payloads, and live log payloads are absent from the generated artifact and receipts.
Live execution remains dry-run by default and requires `BITCODE_V40_REHEARSAL_EXECUTE=1` before delegating to the Vercel Sandbox AssetPack harness.
Staging-testnet receipts bind Supabase project `tkpyosihuouusyaxtbau`, REST host `https://tkpyosihuouusyaxtbau.supabase.co/rest/v1/`, real inference, and database event streaming/readback.

## Gate 10 implementation notes

Gate 10 is closed by `V40PromptBenchmarkSmokeV41Readiness`.
Its generated `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` artifact binds prompt benchmark smoke coverage to ten source-safe rows: benchmark report command execution, deterministic PromptPart smoke execution, deterministic composed Prompt smoke execution, V38 benchmark inventory binding, Reading prompt fixtures, Conversation and tool prompt fixtures, source-safe no-rewrite boundary, V41 prompt-program worklist, workflow wiring, and proof-system wiring.
The `prompt-benchmark:smoke` command uses a local source-safe mock provider and records benchmark scores, fixture roots, and package benchmark report counts without serializing raw prompt text, protected PromptPart source, raw provider responses, credentials, wallet private material, private settlement payloads, or unpaid AssetPack source.
Gate 10 exists to make prompt and PromptPart benchmarking runnable and auditable before V41; every prompt semantic repartition, retitle, rewrite, catalogue improvement, interpolation binding review, and callsite-quality benchmark remains V41 work.
9 changes: 5 additions & 4 deletions BITCODE_SPEC_V40_PARITY_MATRIX.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
- Current canonical/latest target: `V39`
- Prior canonical anchor: `BITCODE_SPEC_V39.md`
- Prior generated proof appendix: `BITCODE_SPEC_V39_PROVEN.md`
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, and later V40 testing artifacts
- Source parity state: V40 testing parity is drafted
- Generated structured artifact inventory: draft `.bitcode/v40-spec-family-report.json`, `.bitcode/v40-canonical-input-report.json`, `.bitcode/v40-test-inventory-coverage-matrix.json`, `.bitcode/v40-unit-coverage-inventory.json`, `.bitcode/v40-api-integration-contracts.json`, `.bitcode/v40-reading-pipeline-integration-coverage.json`, `.bitcode/v40-conversation-terminal-integration.json`, `.bitcode/v40-browser-e2e-visual-proof.json`, `.bitcode/v40-ledger-storage-sync.json`, `.bitcode/v40-local-staging-rehearsal-automation.json`, `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, and later V40 promotion readiness artifacts
- Source parity state: V40 testing parity is closing progressively through generated artifacts

## Purpose

Expand Down Expand Up @@ -36,7 +36,7 @@ This matrix records the V40 testing surfaces that must become promotion-grade be
| Pipeline integration | Primitive and real Reading pipeline implementations are tested | `.bitcode/v40-reading-pipeline-integration-coverage.json` | implemented |
| Conversation/Terminal integration | Conversation handoff, stream logs, Terminal Reading state, and authority boundaries are tested source-safely | `.bitcode/v40-conversation-terminal-integration.json` | implemented |
| Unit coverage | Packages, primitives, isolated implementations, and real implementations have unit coverage | `.bitcode/v40-unit-coverage-inventory.json` | implemented |
| Prompt benchmark smoke | Prompt and PromptPart benchmarks run before V41 | future V40 artifacts | not yet implemented |
| Prompt benchmark smoke | Prompt and PromptPart benchmarks run before V41 | `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json` and `V40PromptBenchmarkSmokeV41Readiness` | implemented |

## V40 implementation checklist

Expand All @@ -51,7 +51,7 @@ This matrix records the V40 testing surfaces that must become promotion-grade be
| Gate 7 | Browser/visual/accessibility/responsive artifact | implemented |
| Gate 8 | Ledger/database/storage/wallet/delivery sync artifact | implemented |
| Gate 9 | Local/staging rehearsal artifact | implemented |
| Gate 10 | Prompt benchmark smoke and V41 readiness artifact | not yet implemented |
| Gate 10 | Prompt benchmark smoke and V41 readiness artifact | implemented |
| Gate 11 | Promotion readiness artifact and workflow | not yet implemented |

Gate 2 implementation evidence: package-backed `V40TestInventoryCoverageMatrix` emits `.bitcode/v40-test-inventory-coverage-matrix.json` and is wired through `check:v40-gate2`.
Expand All @@ -62,6 +62,7 @@ Gate 6 implementation evidence: package-backed `V40ConversationTerminalIntegrati
Gate 7 implementation evidence: package-backed `V40BrowserE2eVisualProof` emits `.bitcode/v40-browser-e2e-visual-proof.json` and is wired through `check:v40-gate7`.
Gate 8 implementation evidence: package-backed `V40LedgerStorageSync` emits `.bitcode/v40-ledger-storage-sync.json` and is wired through `check:v40-gate8`.
Gate 9 implementation evidence: package-backed `V40LocalStagingRehearsalAutomation` emits `.bitcode/v40-local-staging-rehearsal-automation.json` and is wired through `check:v40-gate9`.
Gate 10 implementation evidence: package-backed `V40PromptBenchmarkSmokeV41Readiness` emits `.bitcode/v40-prompt-benchmark-smoke-v41-readiness.json`, runs source-safe PromptPart and composed Prompt smoke receipts through `prompt-benchmark:smoke`, binds V38 benchmark inventory evidence, and is wired through `check:v40-gate10`.

## V40 accepted boundaries

Expand Down
Loading
Loading