Skip to content

Commit 9486249

Browse files
Merge pull request #163 from engineeredsoftware/v41/gate-4-reading-prompt-benchmark-baselines
V41 Gate 4: Reading Prompt Benchmark Baselines
2 parents 0fbb35e + d31b4f2 commit 9486249

19 files changed

Lines changed: 4460 additions & 12 deletions

.bitcode/v41-reading-prompt-benchmark-baselines.json

Lines changed: 2896 additions & 0 deletions
Large diffs are not rendered by default.

.github/workflows/bitcode-canon-quality.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,9 @@ jobs:
268268
if [ -f scripts/check-v41-gate3-registry-interpolation-contracts.mjs ]; then
269269
node scripts/check-v41-gate3-registry-interpolation-contracts.mjs --skip-branch-check --skip-package-tests
270270
fi
271+
if [ -f scripts/check-v41-gate4-reading-prompt-benchmark-baselines.mjs ]; then
272+
node scripts/check-v41-gate4-reading-prompt-benchmark-baselines.mjs --skip-branch-check --skip-package-tests
273+
fi
271274
fi
272275
else
273276
echo "Unexpected BITCODE_SPEC.txt pointer: $POINTER" >&2

.github/workflows/bitcode-gate-quality.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,9 @@ jobs:
397397
if [ -f scripts/check-v41-gate3-registry-interpolation-contracts.mjs ]; then
398398
node scripts/check-v41-gate3-registry-interpolation-contracts.mjs --skip-branch-check --skip-package-tests
399399
fi
400+
if [ -f scripts/check-v41-gate4-reading-prompt-benchmark-baselines.mjs ]; then
401+
node scripts/check-v41-gate4-reading-prompt-benchmark-baselines.mjs --skip-branch-check --skip-package-tests
402+
fi
400403
fi
401404
else
402405
echo "Unexpected BITCODE_SPEC.txt pointer: $POINTER" >&2

BITCODE_SPEC_V41.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,9 @@ The current source-safe contract covers 12 rows, 65 passing source predicates, P
123123

124124
Gate 4 must establish benchmark baselines for Reading prompt surfaces before rewrites.
125125
It covers `ReadNeedComprehensionSynthesis`, `ReadFitsFindingSynthesis`, Need review, Finding Fits, depository query synthesis, many-candidate ranking, AssetPack preview, quote explanation, telemetry summaries, and failure/repair prompts.
126+
Gate 4 is closed by the package-backed `buildV41ReadingPromptBenchmarkBaselines` source, deterministic `.bitcode/v41-reading-prompt-benchmark-baselines.json` artifact, `generate:v41-reading-prompt-benchmark-baselines`, `check:v41-reading-prompt-benchmark-baselines`, `check:v41-gate4`, protocol tests, and gate/canon workflow wiring.
127+
The current source-safe baseline covers 10 rows, 120 passing source predicates, all five Reading UX steps, both Reading pipeline ids, Gate 2 PromptPart/Prompt inventory roots, Gate 3 registry/interpolation roots, V38 benchmark fixture roots, V40 prompt smoke roots, 36 parser target ids, 9 registry contract ids, deterministic baseline scores, and source-safe disclosure tiers.
128+
The artifact may include ids, hashes, counts, fixture ids, metric ids, parser target ids, baseline scores, source path roots, and verdicts, but no raw prompt text, interpolated prompts, raw provider responses, protected source, private context, credentials, or unpaid AssetPack source.
126129

127130
## V41 Gate 5 ReadNeedComprehensionSynthesis Prompt Rewrite And Return-Type Hardening
128131

BITCODE_SPEC_V41_DELTA.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@
77
- Current canonical/latest target: `V40`
88
- Prior canonical anchor: `BITCODE_SPEC_V40.md`
99
- Prior generated proof appendix: `BITCODE_SPEC_V40_PROVEN.md`
10-
- Generated structured artifact inventory: draft `.bitcode/v41-spec-family-report.json`, draft `.bitcode/v41-canonical-input-report.json`, Gate 2 `.bitcode/v41-promptpart-prompt-inventory.json`, Gate 3 `.bitcode/v41-registry-interpolation-contracts.json`, planned benchmark artifacts, planned promotion-readiness artifact, and eventual `BITCODE_SPEC_V41_PROVEN.md` after V41 promotion
11-
- Source parity state: V41 source parity now includes prompt inventory plus registry/interpolation contracts; prompt rewrites are explicitly deferred until catalogue, contract, and benchmark gates close
10+
- Generated structured artifact inventory: draft `.bitcode/v41-spec-family-report.json`, draft `.bitcode/v41-canonical-input-report.json`, Gate 2 `.bitcode/v41-promptpart-prompt-inventory.json`, Gate 3 `.bitcode/v41-registry-interpolation-contracts.json`, Gate 4 `.bitcode/v41-reading-prompt-benchmark-baselines.json`, planned rewrite artifacts, planned promotion-readiness artifact, and eventual `BITCODE_SPEC_V41_PROVEN.md` after V41 promotion
11+
- Source parity state: V41 source parity now includes prompt inventory, registry/interpolation contracts, and Reading benchmark baselines; prompt rewrites are explicitly deferred until catalogue, contract, and benchmark gates close
1212
- Scope: V41 draft delta for Prompt and PromptPart excellence over promoted V40
1313

1414
## Why V41 exists
@@ -45,6 +45,8 @@ It preserves V42 as subsequent reliable MVP experience planning while keeping V4
4545
### Gate 4: Reading Pipeline Prompt Benchmark Baselines
4646

4747
Create source-safe baseline benchmarks for Reading prompt surfaces before rewriting them.
48+
Gate 4 is package-backed by `buildV41ReadingPromptBenchmarkBaselines`, emits `.bitcode/v41-reading-prompt-benchmark-baselines.json`, and covers 10 source-safe baseline rows with 120 passing predicates across Read Request-to-Need synthesis, Need review/resynthesis, Need measurement and quote posture, Finding Fits query synthesis, many-candidate ranking, AssetPack synthesis context, source-safe AssetPack preview, settlement/rights/delivery, telemetry summaries, and failure/repair prompts.
49+
It binds Gate 2 PromptPart/Prompt inventory, Gate 3 registry/interpolation contracts, V38 benchmark fixtures, V40 prompt smoke readiness, both Reading pipelines, all five Reading UX steps, parser target ids, disclosure tiers, and source-safe score metadata without serializing raw prompt text, interpolated prompts, provider responses, protected source, private context, credentials, or unpaid AssetPack source.
4850

4951
### Gate 5: ReadNeedComprehensionSynthesis Prompt Rewrite And Return-Type Hardening
5052

@@ -71,6 +73,7 @@ Bind all V41 prompt artifacts, tests, workflow checks, generated proof support,
7173
- No prompt content is rewritten during Gate 1.
7274
- New commercial product behavior is deferred unless needed to make prompt-program tests truthful.
7375
- Production-mainnet value-bearing prompt rehearsals remain opt-in.
76+
- V43+ agentic enterprise depositing is deferred: deposit AssetPack option synthesis, repository-installed deposit agents, sub-critical/positive-ROI IP filtering, `/read` and `/deposit` route separation, and `/exchange` to `/packs` renaming are planning notes only until a later version explicitly opens that scope.
7477

7578
## Pre-Implementation Sequence
7679

BITCODE_SPEC_V41_NOTES.md

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
- Canonical pointer: `BITCODE_SPEC.txt` -> `V40`
99
- Prior canonical anchor: `BITCODE_SPEC_V40.md`
1010
- Prior generated proof appendix: `BITCODE_SPEC_V40_PROVEN.md`
11-
- Generated structured artifact inventory: draft `.bitcode/v41-spec-family-report.json`, draft `.bitcode/v41-canonical-input-report.json`, Gate 2 `.bitcode/v41-promptpart-prompt-inventory.json`, Gate 3 `.bitcode/v41-registry-interpolation-contracts.json`, planned benchmark artifacts, planned promotion-readiness artifact, and eventual `BITCODE_SPEC_V41_PROVEN.md` after V41 promotion
11+
- Generated structured artifact inventory: draft `.bitcode/v41-spec-family-report.json`, draft `.bitcode/v41-canonical-input-report.json`, Gate 2 `.bitcode/v41-promptpart-prompt-inventory.json`, Gate 3 `.bitcode/v41-registry-interpolation-contracts.json`, Gate 4 `.bitcode/v41-reading-prompt-benchmark-baselines.json`, planned rewrite artifacts, planned promotion-readiness artifact, and eventual `BITCODE_SPEC_V41_PROVEN.md` after V41 promotion
1212
- Source parity state: V41 notes track prompt-program planning, catalogue, benchmark, rewrite, telemetry, and promotion parity over active V40
1313
- Draft target posture: V41 Gate 1 creates the full SPEC, DELTA, NOTES, and PARITY family.
1414
- Scope: Prompt and PromptPart excellence over the promoted V40 exhaustive testing base.
@@ -33,12 +33,39 @@ Depositing must minimize the path to admitting source material and later receivi
3333
Reading must minimize the path from Read Request to synthesized Need review/resynthesis, Finding Fits, source-safe AssetPack preview, BTD/BTC purchase and settlement, and repository delivery.
3434
V42 must keep depositor compensation, reader purchase, BTD/AssetPack rights, and repository delivery visible in the shortest path without widening V41 beyond Prompt and PromptPart excellence.
3535

36+
## V43+ agentic depositing roadmap note
37+
38+
V43 or a later explicitly opened version should evolve the deposit side from manual source admission into an agentic AssetPack option experience for enterprises that own connected codebases.
39+
The central object is a deposit AssetPack option: an unminted AssetPack in all but BTD, synthesized from proprietary or otherwise non-public source material and held for enterprise review before Depository admission.
40+
BTD remains minted only when an industrial Need-Fit is made for a Reader; the deposit-side option is not BTD yet, but it is shaped as an AssetPack candidate so that future Finding Fits can search, rank, synthesize from, and compensate it cleanly.
41+
42+
Bitcode Agents installed in an enterprise repository should compare three live contexts:
43+
44+
- the enterprise's connected codebase and its evolving source/measurement state;
45+
- the existing Bitcode Depository and pre-existing deposits/AssetPacks;
46+
- current and historical Reading activity that indicates demand for technical knowledge.
47+
48+
The deposit-side synthesis pipeline should propose AssetPack options only when the option appears sub-critical to the enterprise, not likely to leak strategic or protected IP the enterprise would not sell, and likely positive ROI after considering development cost, uniqueness, expected demand, and likely future BTD mint value.
49+
Those criteria must be configurable and promptable by the enterprise, but the default UX should minimize manual decision-making by surfacing clear approve/reject options with source-safe measurements, demand evidence, criticality assessment, ROI posture, and compensation route expectations.
50+
Approved options enter the Bitcode Depository and become eligible for Read-Need Finding Fits; rejected options remain out of the Depository and must not be searched or disclosed.
51+
52+
That version should also split the product navigation now concentrated in `/terminal`.
53+
`/read` should own Read Request submission, synthesized Need review/resynthesis, Finding Fits, source-safe AssetPack preview, purchase, settlement, and repository delivery.
54+
`/deposit` should own connected repository state, agentic deposit AssetPack option synthesis, manual reruns reflecting the latest connected repositories and Bitcode state, approve/reject review, Depository admission, and later compensation readback.
55+
`/exchange` should be renamed to `/packs` across routes, code naming, docs, and operator vocabulary, because the product object is AssetPacks and BTD rights rather than a generic exchange abstraction.
56+
3657
## Gate 3 registry and interpolation contract note
3758

3859
Gate 3 makes the prompt registry call chain source-checkable before prompt rewrites.
3960
The package-backed contract currently emits 12 rows and 65 source predicates for Prompt registry totality, TemplatedPromptPart interpolation, PTRR agent and step prompt composition, FailsafeGenerationSequence context handling, ThricifiedGeneration final prompt resolution, execution ancestry overlays, tool doc-code prompt injection, Reading parser targets, Finding Fits search and embedding contracts, AssetPack synthesis and finishing parser targets, and Gate 2 inventory binding.
4061
The artifact records ids, source paths, hashes, predicate verdicts, parser target ids, interpolation key ids, and ancestry frame ids only; it does not serialize raw prompt text, provider responses, private context, credentials, or unpaid AssetPack source.
4162

63+
## Gate 4 Reading benchmark baseline note
64+
65+
Gate 4 makes Reading prompt quality source-checkable before prompt rewrites.
66+
The package-backed baseline currently emits 10 rows and 120 source predicates for Read Request-to-Need synthesis, Need review/resynthesis, Need measurement and quote posture, Finding Fits query synthesis, many-candidate ranking, AssetPack synthesis context, source-safe AssetPack preview, settlement/rights/delivery, telemetry summaries, and failure/repair prompts.
67+
The artifact binds both Reading pipelines, all five Reading UX steps, Gate 2 prompt inventory, Gate 3 registry/interpolation contracts, V38 benchmark fixtures, V40 prompt smoke readiness, parser target ids, and source-safe score metadata only; it does not serialize raw prompt text, interpolated prompts, provider responses, private context, protected source, credentials, or unpaid AssetPack source.
68+
4269
## Concise current-system reading
4370

4471
Bitcode is active at V40.

BITCODE_SPEC_V41_PARITY_MATRIX.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@
77
- Current canonical/latest target: `V40`
88
- Prior canonical anchor: `BITCODE_SPEC_V40.md`
99
- Prior generated proof appendix: `BITCODE_SPEC_V40_PROVEN.md`
10-
- Generated structured artifact inventory: draft `.bitcode/v41-spec-family-report.json`, draft `.bitcode/v41-canonical-input-report.json`, Gate 2 `.bitcode/v41-promptpart-prompt-inventory.json`, Gate 3 `.bitcode/v41-registry-interpolation-contracts.json`, planned benchmark artifacts, planned promotion-readiness artifact, and eventual `BITCODE_SPEC_V41_PROVEN.md` after V41 promotion
11-
- Source parity state: Gate 1 closes spec, roadmap, docs, package script, workflow posture, and checker parity; Gate 2 closes source-safe prompt inventory parity; Gate 3 closes source-safe registry/interpolation contract parity; later gates own rewrite, benchmark, telemetry, and promotion parity
10+
- Generated structured artifact inventory: draft `.bitcode/v41-spec-family-report.json`, draft `.bitcode/v41-canonical-input-report.json`, Gate 2 `.bitcode/v41-promptpart-prompt-inventory.json`, Gate 3 `.bitcode/v41-registry-interpolation-contracts.json`, Gate 4 `.bitcode/v41-reading-prompt-benchmark-baselines.json`, planned rewrite artifacts, planned promotion-readiness artifact, and eventual `BITCODE_SPEC_V41_PROVEN.md` after V41 promotion
11+
- Source parity state: Gate 1 closes spec, roadmap, docs, package script, workflow posture, and checker parity; Gate 2 closes source-safe prompt inventory parity; Gate 3 closes source-safe registry/interpolation contract parity; Gate 4 closes source-safe Reading benchmark baseline parity; later gates own rewrite, benchmark, telemetry, and promotion parity
1212
- Scope: V41 draft parity ledger for Prompt and PromptPart excellence
1313
- Last fully realized canonical target preserved in source: `V40`
1414

@@ -36,7 +36,7 @@ This matrix records the prompt-program surfaces that must become promotion-grade
3636
| Canon workflow | Canon quality knows active V40 and draft V41 Gate 1 | `.github/workflows/bitcode-canon-quality.yml` | drafted |
3737
| Prompt inventory | Every raw PromptPart and composed Prompt catalogued | `.bitcode/v41-promptpart-prompt-inventory.json` | implemented |
3838
| Registry/interpolation | Registry composition and interpolation contracts proven | `.bitcode/v41-registry-interpolation-contracts.json` | implemented |
39-
| Reading baselines | Reading prompt benchmarks captured before rewrites | planned V41 Gate 4 artifact | pending |
39+
| Reading baselines | Reading prompt benchmarks captured before rewrites | `.bitcode/v41-reading-prompt-benchmark-baselines.json` | implemented |
4040
| ReadNeed rewrite | ReadNeedComprehensionSynthesis prompts rewritten and type-hardened | planned V41 Gate 5 artifact | pending |
4141
| Finding Fits rewrite | ReadFitsFindingSynthesis prompts rewritten for search and AssetPack context | planned V41 Gate 6 artifact | pending |
4242
| Conversation/tool/interface rewrite | Non-Reading prompt surfaces brought to same standard | planned V41 Gate 7 artifact | pending |
@@ -50,7 +50,7 @@ This matrix records the prompt-program surfaces that must become promotion-grade
5050
| Gate 1 | Open V41 family, roadmap, docs, workflow posture, package script, and checker | closed |
5151
| Gate 2 | PromptPart and Prompt inventory artifact | implemented |
5252
| Gate 3 | Registry composition and interpolation artifact | implemented |
53-
| Gate 4 | Reading prompt benchmark baseline artifact | pending |
53+
| Gate 4 | Reading prompt benchmark baseline artifact | implemented |
5454
| Gate 5 | ReadNeedComprehensionSynthesis rewrite artifact | pending |
5555
| Gate 6 | ReadFitsFindingSynthesis rewrite artifact | pending |
5656
| Gate 7 | Conversation, tool, and interface prompt rewrite artifact | pending |
@@ -71,6 +71,10 @@ V41 Gate 3 may catalogue registry ids, interpolation key ids, missing-key behavi
7171
It may not serialize raw prompt text, raw provider responses, protected source, private context, credentials, settlement-private payloads, or unpaid AssetPack source.
7272
It closes prompt rewrites' precondition that prompt composition and interpolation contracts are observable before semantic PromptPart or Prompt rewrite work proceeds.
7373

74+
V41 Gate 4 may catalogue Reading benchmark baseline ids, fixture ids, metric ids, parser target ids, source path roots, predicate verdicts, deterministic baseline scores, inventory bindings, registry/interpolation bindings, and source-safe disclosure tiers for `ReadNeedComprehensionSynthesis` and `ReadFitsFindingSynthesis`.
75+
It may not serialize raw prompt text, interpolated prompts, provider responses, protected source, private context, credentials, settlement-private payloads, or unpaid AssetPack source.
76+
It closes Reading prompt rewrites' benchmark precondition by proving source-safe baselines for all five Reading UX steps before semantic PromptPart or Prompt rewrite work starts.
77+
7478
## V41 completion condition
7579

7680
V41 closes when prompt-program quality is fully specified, implemented, benchmarked, source-safe, tested, documented, generated, workflow-checked, and promotion-ready across Reading, Conversation, tool-definition, and interface inference surfaces.

README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,24 @@ agent/step composition, Failsafe and Thricified prompt resolution, execution
3333
ancestry overlays, tool doc-code prompt injection, Reading parser targets,
3434
Finding Fits search contracts, AssetPack synthesis/finishing parser targets,
3535
and Gate 2 inventory binding with `check:v41-gate3`.
36+
V41 Gate 4 adds the package-backed `buildV41ReadingPromptBenchmarkBaselines`
37+
and generated source-safe artifact
38+
`.bitcode/v41-reading-prompt-benchmark-baselines.json`. The baseline currently
39+
covers 10 Reading prompt benchmark rows, 120 source predicates, all five Reading
40+
UX steps, both Reading pipelines, Gate 2 inventory, Gate 3 registry contracts,
41+
V38 benchmark fixtures, V40 prompt smoke readiness, parser targets, and
42+
source-safe disclosure tiers with `check:v41-gate4`.
3643
V42 is now roadmapped as the next MVP experience version: shortest-path
3744
Depositing with later BTC compensation, shortest-path Reading through
3845
Need review/resynthesis, Finding Fits, source-safe AssetPack preview,
3946
BTD/BTC settlement, repository delivery, and an AI-reading dominant
4047
demonstration whose AssetPack measurably improves an AI system beyond
4148
public-data-only performance.
49+
V43+ is roadmapped as the later agentic depositing evolution: repository
50+
agents synthesize deposit AssetPack options from connected enterprise code,
51+
Depository state, and Reading demand; enterprises approve or reject
52+
sub-critical positive-ROI options; `/terminal` separates into `/read` and
53+
`/deposit`; and `/exchange` is renamed to `/packs` across product naming.
4254

4355
Exchange is inherited V36 canon: market-wide activity master-detail, buy/sell/
4456
bid/ask/cancel/accept/settle/history flows, AssetPack range trading,

0 commit comments

Comments
 (0)