Skip to content

Commit aa60bf4

Browse files
wip v28
1 parent ab6ca95 commit aa60bf4

14 files changed

Lines changed: 751 additions & 159 deletions

File tree

BITCODE_SPEC_V28_NOTES.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,50 @@ it checks Deposit before Read, Read before Fit, repository/branch/commit
116116
alignment, frontier/mock leakage, and visibility of proof or measurement
117117
posture in the execution rows.
118118

119+
## May 16 Pipeline Harness Telemetry Boundary
120+
121+
Read/Fit review now requires proof that the deployed AssetPack pipeline host
122+
actually executed, not only that Terminal wrote a Fit posture row.
123+
124+
The active harness target is Vercel Sandbox: ephemeral Amazon Linux microVMs
125+
with Node/Python runtimes, isolated filesystem/process/network scope, exported
126+
artifacts, and database readback. Local development authenticates through
127+
`vercel link && vercel env pull` OIDC credentials; deployed Vercel code receives
128+
OIDC automatically; external hosts need `VERCEL_TOKEN`, `VERCEL_TEAM_ID`, and
129+
`VERCEL_PROJECT_ID`.
130+
131+
The minimum commercial telemetry record for a Read/Fit pipeline execution is:
132+
133+
- one run row in `pipeline_runs` or `deliverable_pipeline_runs`;
134+
- event rows in `execution_events`, `stream_logs`, or
135+
`deliverable_pipeline_events`;
136+
- SDIVF phase rows in `deliverable_pipeline_phase_delegations`;
137+
- PTRR agent-step rows in `deliverable_pipeline_agent_steps`;
138+
- generation rows in `deliverable_pipeline_generations`, including
139+
interpolated messages/prompts when available, model provider/name, raw
140+
response, parsed/typed output, usage tokens, phase, agent, step, failsafe, and
141+
generation context;
142+
- tool rows in `deliverable_pipeline_tool_executions`, including input, output,
143+
error, timing, phase, and agent context;
144+
- fit/depository output carrying `depositorySearch`, candidate ranking,
145+
`fitResult`, `queryRoot`, `rankingRoot`, selected candidate asset ids, and the
146+
embedding policy.
147+
148+
Depository vector recall remains one canonical vector space until explicitly
149+
remigrated: OpenAI `text-embedding-3-small`, float embeddings, 1536 dimensions,
150+
`deliverable_vectors.embedding vector(1536)`, `ivfflat`,
151+
`vector_cosine_ops`, and `match_deliverable_vectors` cosine similarity. Any
152+
different embedding model, dimension, metric, table, or RPC is a blocker unless
153+
all stored candidate vectors and query vectors for the compared space were
154+
regenerated under the same policy.
155+
156+
Saved query `v28_qa_terminal_07_pipeline_harness_after_fit` is the pipeline
157+
harness gate. `blocker:pipeline_harness_run_missing`,
158+
`blocker:pipeline_event_telemetry_missing`, and
159+
`blocker:pipeline_phase_trace_missing` all keep the commercial Fit in
160+
blocked-readiness even when Deposit, Read admission, and source-bound Fit
161+
posture rows exist.
162+
119163
## May 15 Terminal Terminology Closure
120164

121165
V28 now retires user-facing Deposit/Depositing and Read/Reading language in favor of Deposit/Depositing and Read/Reading.

BITCODE_V28_QA.md

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -904,6 +904,18 @@ First-run execution boundary:
904904
storage: `depository/search.result`, `depository/search.candidateRanking`,
905905
`depository/search.selectedCandidates`, `depository/search.embeddingPolicy`,
906906
`fit.result`, `fit.resultState`, and `fit.resultReasons`.
907+
- The deployed pipeline host must also emit structured execution rows for the
908+
run, event stream, SDIVF phase delegations, PTRR agent steps, model
909+
generations, and tool executions. The minimum database readback surface is
910+
`deliverable_pipeline_runs`, `deliverable_pipeline_events`,
911+
`deliverable_pipeline_phase_delegations`,
912+
`deliverable_pipeline_agent_steps`, `deliverable_pipeline_generations`, and
913+
`deliverable_pipeline_tool_executions`.
914+
- Generation telemetry must preserve the interpolated model input messages when
915+
available, the raw generation payload, the parsed/typed response shape, model
916+
provider/name, usage tokens, phase, agent, step, failsafe, and generation
917+
context. Missing prompts, missing parsed outputs, or uncorrelated
918+
generation/tool rows are blockers for debugging live Read/Fit execution.
907919
- Depository vector recall uses `text-embedding-3-small` by default with
908920
`encoding_format='float'`, `dimensions=1536`, Supabase
909921
`deliverable_vectors.embedding vector(1536)`, `ivfflat`,
@@ -1007,9 +1019,14 @@ Manual steps:
10071019
`selectedCandidateAssetIds` are present in the pipeline output or execution
10081020
storage summary. Also verify `embeddingPolicy.model`,
10091021
`embeddingPolicy.dimensions`, and `embeddingPolicy.vectorStore.rpc`.
1010-
11. Rerun `v28_qa_terminal_02_activity_after_write` and
1022+
11. Run saved query `v28_qa_terminal_07_pipeline_harness_after_fit`. It must
1023+
show at least one recent pipeline run or deliverable pipeline run, event
1024+
telemetry, phase trace rows, agent-step rows, and generation/tool rows
1025+
before a `worthy_fit` or `no_worthy_fit` classification can graduate from
1026+
posture into commercial result review.
1027+
12. Rerun `v28_qa_terminal_02_activity_after_write` and
10111028
`v28_qa_terminal_03_btd_ledger_after_terminal`.
1012-
12. Paste screenshots, Network payload summaries, Vercel logs for the same
1029+
13. Paste screenshots, Network payload summaries, Vercel logs for the same
10131030
timestamps, and all three query outputs.
10141031
10151032
Pass criteria:
@@ -1029,6 +1046,12 @@ Pass criteria:
10291046
- Real `worthy_fit` or `no_worthy_fit` classification is backed by deployed
10301047
pipeline execution rows, events, logs, depository search evidence, candidate
10311048
ranking roots, embedding policy, and result evidence.
1049+
- Query `v28_qa_terminal_07_pipeline_harness_after_fit` reports
1050+
`pipeline_harness_ready_for_result_review`; a
1051+
`blocker:pipeline_harness_run_missing`,
1052+
`blocker:pipeline_event_telemetry_missing`, or
1053+
`blocker:pipeline_phase_trace_missing` result means the Fit must remain
1054+
blocked-readiness.
10321055
- Settlement, finality, BTC fee, BTD range, or ledger anchor claims appear only
10331056
when query 03 shows matching projection rows; otherwise the Terminal result
10341057
labels the exact blocked-readiness state.

packages/pipeline-hosts/README.md

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,8 @@ leaving enough telemetry for SQL readback.
99

1010
- Firecracker microVM isolation on Amazon Linux 2023.
1111
- Default working directory: `/vercel/sandbox`.
12-
- Default runtime: `node24`; stable documented runtime options are `node24`,
13-
`node22`, and `python3.13`.
12+
- Default runtime: `node24`; documented runtime options include `node26`,
13+
`node24`, `node22`, and `python3.13`.
1414
- The sandbox user is `vercel-sandbox` and can use `sudo`.
1515
- Filesystem state is ephemeral. Evidence must be exported before stop, then
1616
persisted to Bitcode storage or database projections.
@@ -39,6 +39,22 @@ The exported evidence must include the AssetPack embedding policy
3939
(`text-embedding-3-small`, `1536` dimensions, cosine
4040
`match_deliverable_vectors`) so SQL readback can detect vector-space drift.
4141

42+
Structured database telemetry is part of the harness contract. A real Read/Fit
43+
pipeline run must write the deliverable hierarchy:
44+
`deliverable_pipeline_runs`, `deliverable_pipeline_events`,
45+
`deliverable_pipeline_phase_delegations`,
46+
`deliverable_pipeline_agent_steps`, `deliverable_pipeline_generations`, and
47+
`deliverable_pipeline_tool_executions`. Generation rows should retain the
48+
interpolated model messages when available, raw and parsed output, model
49+
identity, token usage, and phase/agent/step context.
50+
51+
For live model execution, the sandbox runner also needs `OPENAI_API_KEY` in the
52+
trusted command environment. For local Vercel Sandbox creation, either pull
53+
`VERCEL_OIDC_TOKEN` with `vercel link && vercel env pull`, or provide the access
54+
token tuple `VERCEL_TOKEN`, `VERCEL_TEAM_ID`, and `VERCEL_PROJECT_ID`. Deployed
55+
Vercel code should use automatic OIDC rather than storing a Vercel token when
56+
possible.
57+
4258
## Live QA
4359

4460
Link a Vercel project and pull local OIDC credentials:
@@ -73,6 +89,16 @@ BITCODE_SANDBOX_DEPOSIT_HAS_MEASUREMENT=1 \
7389
pnpm -C packages/pipeline-hosts run qa:asset-pack:sandbox
7490
```
7591

92+
After the run, execute:
93+
94+
```bash
95+
psql "$SUPABASE_DB_URL" -v ON_ERROR_STOP=1 \
96+
-f supabase/queries/v28_qa_terminal_07_pipeline_harness_after_fit.sql
97+
```
98+
99+
The harness is not commercially reviewable until this query reports pipeline
100+
run/event/phase/agent/generation/tool visibility rather than a blocker state.
101+
76102
Only pass secrets through `BITCODE_SANDBOX_ENV_KEYS` when the sandbox code path
77103
is trusted and its network policy is understood. Prefer credential brokering or
78104
strict allowlists before running untrusted generated code.

packages/pipelines-generics/src/execution/PipelineExecutor.ts

Lines changed: 82 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -54,15 +54,38 @@ export class PipelineExecutor {
5454
if (!agent) {
5555
throw new Error(`Agent not found: ${agentName}`);
5656
}
57-
58-
// Note: Agent start/complete events are emitted at step level via store()
57+
const phase = String(this.execution.get('phase', 'current') || 'setup');
58+
const step = 'try';
59+
this.execution.store('agent', 'name', agentName);
60+
this.execution.store('step', 'name', step);
61+
this.execution.store(`agent:${agentName}`, 'start', {
62+
phase,
63+
currentPhase: phase,
64+
agent: agentName,
65+
currentAgent: agentName,
66+
step,
67+
currentStep: step,
68+
status: 'running',
69+
input: summarizeValue(input),
70+
startedAt: new Date().toISOString(),
71+
} as any);
5972

6073
// Execute agent with error envelope
6174
let output: any;
6275
try {
6376
output = await agent(input, this.execution);
6477
} catch (error) {
65-
// Non-fatal: streaming for errors handled elsewhere via status/error writes
78+
this.execution.store(`agent:${agentName}`, 'complete', {
79+
phase,
80+
currentPhase: phase,
81+
agent: agentName,
82+
currentAgent: agentName,
83+
step,
84+
currentStep: step,
85+
status: 'failed',
86+
error: summarizeError(error),
87+
completedAt: new Date().toISOString(),
88+
} as any);
6689
throw error;
6790
}
6891

@@ -72,14 +95,34 @@ export class PipelineExecutor {
7295
this.execution.store('pipeline/short-circuit', 'signal', output.signal as any);
7396
this.execution.store('pipeline/short-circuit', 'agent', agentName);
7497
this.execution.store('pipeline/short-circuit', 'timestamp', Date.now());
75-
// Step-level store emits handle agent-complete with step context
98+
this.execution.store(`agent:${agentName}`, 'complete', {
99+
phase,
100+
currentPhase: phase,
101+
agent: agentName,
102+
currentAgent: agentName,
103+
step,
104+
currentStep: step,
105+
status: 'completed',
106+
shortCircuited: true,
107+
output: summarizeValue(output),
108+
completedAt: new Date().toISOString(),
109+
} as any);
76110

77111
// Throw short-circuit error for phase runner to catch
78112
throw new ShortCircuitError(output.signal!);
79113
}
80114

81-
// Emit end
82-
// Step-level store emits handle agent-complete with step context
115+
this.execution.store(`agent:${agentName}`, 'complete', {
116+
phase,
117+
currentPhase: phase,
118+
agent: agentName,
119+
currentAgent: agentName,
120+
step,
121+
currentStep: step,
122+
status: 'completed',
123+
output: summarizeValue(output),
124+
completedAt: new Date().toISOString(),
125+
} as any);
83126

84127
// Return normal result
85128
return output.result || output;
@@ -183,3 +226,36 @@ export function createPhaseRunner(config: PhaseConfig): Executor<any, PhaseResul
183226
return await executor.executePhase(config);
184227
};
185228
}
229+
230+
function summarizeError(error: unknown): Record<string, unknown> {
231+
if (error instanceof Error) {
232+
return {
233+
name: error.name,
234+
message: error.message,
235+
stack: error.stack ? error.stack.split('\n').slice(0, 6).join('\n') : undefined,
236+
};
237+
}
238+
return { message: String(error) };
239+
}
240+
241+
function summarizeValue(value: unknown): unknown {
242+
try {
243+
if (value == null) return value;
244+
if (typeof value === 'string') {
245+
return value.length > 500 ? `${value.slice(0, 500)}... [truncated]` : value;
246+
}
247+
if (Array.isArray(value)) {
248+
return { type: 'array', length: value.length, sample: summarizeValue(value[0]) };
249+
}
250+
if (typeof value === 'object') {
251+
const objectValue = value as Record<string, unknown>;
252+
const keys = Object.keys(objectValue);
253+
const sample: Record<string, unknown> = {};
254+
for (const key of keys.slice(0, 8)) sample[key] = summarizeValue(objectValue[key]);
255+
return { type: 'object', keys: keys.slice(0, 20), sample };
256+
}
257+
return value;
258+
} catch {
259+
return '[unserializable]';
260+
}
261+
}

packages/pipelines-generics/src/execution/__tests__/events.unit.test.ts

Lines changed: 22 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,18 @@
22
// @ts-nocheck
33
import { PipelineExecution } from '../PipelineExecution';
44
import { PipelineExecutor } from '../PipelineExecutor';
5+
import { ExecutionStreamAdapter } from '@bitcode/execution-generics';
6+
import { Streamer } from '@bitcode/streams';
57

68
describe('PipelineExecutor event emission (unit)', () => {
79
it('emits phase and agent events in expected order', async () => {
810
const exec = new PipelineExecution('pipeline:unit');
911
const events: any[] = [];
10-
exec.store('execution', 'dataStream', {
11-
writeData: async (data: string) => {
12-
try { events.push(JSON.parse(data)); } catch { /* ignore */ }
13-
}
12+
const streamer = new Streamer({ streamId: 'pipeline:unit', userId: 'unit-user' });
13+
streamer.subscribe((event) => {
14+
events.push(event);
1415
});
16+
ExecutionStreamAdapter.registerStreamer(exec.id, streamer);
1517

1618
// Register two trivial agents
1719
const agentA = async (input: any) => ({ ok: 'A' });
@@ -31,15 +33,24 @@ describe('PipelineExecutor event emission (unit)', () => {
3133
});
3234

3335
expect(res.success).toBe(true);
34-
const types = events.map(e => e.type + (e.phase ? ':'+e.phase : '') + (e.agent ? ':'+e.agent : '') + (e.status ? ':'+e.status : ''));
36+
await new Promise((resolve) => setTimeout(resolve, 20));
37+
ExecutionStreamAdapter.unregisterStreamer(exec.id);
38+
const types = events
39+
.filter((event) => ['phase-start', 'phase-complete', 'agent-start', 'agent-complete'].includes(event.type))
40+
.map((event) => [
41+
event.type,
42+
event.executionState?.phase || event.data?.phase || event.phase || '',
43+
event.executionState?.agent || event.data?.agent || event.agent || '',
44+
event.data?.status || event.status || '',
45+
].join(':'));
3546
// Order: phase:start -> agentA:start -> agentA:end -> agentB:start -> agentB:end -> phase:end
3647
expect(types).toEqual([
37-
'phase:setup:start',
38-
'agent:unit:agentA:start',
39-
'agent:unit:agentA:end',
40-
'agent:unit:agentB:start',
41-
'agent:unit:agentB:end',
42-
'phase:setup:end',
48+
'phase-start:setup::',
49+
'agent-start:setup:unit:agentA:running',
50+
'agent-complete:setup:unit:agentA:completed',
51+
'agent-start:setup:unit:agentB:running',
52+
'agent-complete:setup:unit:agentB:completed',
53+
'phase-complete:setup::',
4354
]);
4455
});
4556
});

0 commit comments

Comments
 (0)