Skip to content

Commit 8474287

Browse files
geraldarthurdavisgeraldarthurdavis
committed
V48 (specification-implementation): Document and wire Agent/PTRR tool interpolations
Add TOOLS-IN-PTRR guide; auto-inject DocCode tool docs and usedTools results into Thinkings prompts; normalize useTools selection shapes; expand terminology and package READMEs for parameters/results contracts.
1 parent 4ec85e0 commit 8474287

10 files changed

Lines changed: 567 additions & 19 deletions

File tree

‎FAMILIARIZATION.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -976,8 +976,16 @@ in new code and docs. Deeper product law lives in the SPEC; packaging law in
976976
| **StitchUntilComplete** | FailsafeGeneration kind: repair incomplete/truncated structured output. |
977977
| **StructuredOutput** | ThinkingsGeneration kind: emit schema-shaped final output. |
978978
| **ThinkingsGeneration** | Child kinds of each failsafe: Reason → Judge → StructuredOutput. |
979-
| **Tool** | Callable capability attached to agents/pipelines (`tools-generics` / `generic-tools/*`). |
980-
| **Tool postprocess** | Tools run after failsafes within a PTRR step architecture. |
979+
| **Tool** | Callable capability (`tools-generics` / `generic-tools/*`); Agent uses `ExecutionTool` + registry. |
980+
| **Tool postprocess** | After Failsafe×Thinkings: `factoryToolsExecution` if `output.useTools?.length`. |
981+
| **useTools** | Planned invocations from structured output: `{ name, input, reason }[]`. |
982+
| **usedTools** | Results: `{ tool, input?, output?, error? }[]` for telemetry + later steps. |
983+
| **Doc interpolation** | Usable tools' DocCodeToolPrompt text → prompt path `auto:tools_doc_code_tools`. |
984+
| **Results interpolation** | Prior usedTools → prompt path `auto:tools_results`. |
985+
| **formatUsableTools** | Renders DocCodeToolPrompt docs for an array of Tool instances. |
986+
| **AgentToolsRegistry** | Hierarchical tool lookup (`getTool` / `getUsableTools` / `restrictTo`). |
987+
988+
Guide: `packages/agent-generics/TOOLS-IN-PTRR.md`.
981989

982990
### 12.5 Pipelines, hosts, measurements, AssetPack layers
983991

‎internal-docs/TERMINOLOGY.md‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,13 +50,26 @@ export enum ThinkingsGeneration {
5050
}
5151
```
5252

53-
#### Tools (1)
54-
- `tools_execution` — after all failsafes
53+
#### Tools (postprocess, not a Failsafe/Thinkings generation)
54+
- `tools_execution` — after all failsafes, **once**, if structured output includes `useTools`
5555

56-
**Total per PTRR step**: 3 FailsafeGenerations × Thinkings composition + tools postprocess.
56+
**Total per PTRR step**: 3 FailsafeGenerations × Thinkings composition + optional tools postprocess.
5757

5858
Architecture interface: `PTRRStepGenerationArchitecture`.
5959

60+
### Tools: parameters, docs, results
61+
62+
| Term | Meaning |
63+
| --- | --- |
64+
| **Usable tools** | Tools visible via `AgentToolsRegistry.getUsableTools()` (hierarchy). |
65+
| **Doc interpolation** | `formatUsableTools` → `auto:tools_doc_code_tools` on Thinkings prompts. |
66+
| **useTools** | LLM selection: `{ name, input, reason }[]` on step structured output. |
67+
| **usedTools** | Execution results: `{ tool, input?, output?, error? }[]`. |
68+
| **Results interpolation** | Prior `usedTools` → `auto:tools_results` on later generations. |
69+
| **DocCodeToolPrompt** | Prompt carrier for purpose/capabilities/**parameters**/**output** sections. |
70+
71+
Full guide: `packages/agent-generics/TOOLS-IN-PTRR.md`.
72+
6073
### Legacy names (do not use in new code)
6174

6275
| Legacy | Prefer |

‎packages/agent-generics/README.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,17 @@ product / generic-agent-* # specialized agents
1616
```
1717

1818
Within each PTRR step: **FailsafeGeneration** ×3 (each runs **ThinkingsGeneration**
19-
Reason→Judge→StructuredOutput) + tools postprocess.
19+
Reason→Judge→StructuredOutput) + **tools postprocess** (if `output.useTools`).
20+
21+
### Tools (parameters, docs, results)
22+
23+
See **[TOOLS-IN-PTRR.md](./TOOLS-IN-PTRR.md)** for the full contract:
24+
25+
1. Register tools on `AgentExecution.tools` (or parent pipeline registry).
26+
2. Doc-code docs auto-interpolate as `auto:tools_doc_code_tools` before Thinkings LLM calls.
27+
3. Model selects `useTools: [{ name, input, reason }]`.
28+
4. `factoryToolsExecution` runs `getTool(name).execute(input)` → `usedTools`.
29+
5. Prior `usedTools` auto-interpolate as `auto:tools_results` on later generations.
2030

2131
Legacy `*MetaSubStep` / `SubStep` names are BC aliases only — prefer
2232
`FailsafeGeneration` / `ThinkingsGeneration` / `GenerationExecution`.

‎packages/agent-generics/TOOLS-IN-PTRR.md‎

Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
# Tools in the Agent primitive + PTRR
2+
3+
How tools are **documented**, **selected**, **parameterized**, **executed**, and
4+
**result-interpolated** for Bitcode agents.
5+
6+
Related packages:
7+
8+
| Package | Role |
9+
| --- | --- |
10+
| `@bitcode/tools-generics` | `Tool`, DocCodeToolPrompt, `formatUsableTools` |
11+
| `@bitcode/agent-generics` | AgentExecution tools registry, PTRR step postprocess, interpolations |
12+
| `@bitcode/generic-agents-ptrr` | PTRRAgent base (Plan→Try→Refine→Retry) |
13+
| `@bitcode/generic-doc-comments-doc-code` | Build-time `@doc-code-tool` → prompt attachment |
14+
15+
---
16+
17+
## 1. End-to-end lifecycle
18+
19+
```
20+
Register tools on AgentExecution.tools (or parent pipeline registry)
21+
↓
22+
PTRR step starts → store tools.usable = Object.keys(getUsableTools())
23+
↓
24+
FailsafeGeneration ×3 (PCC → ChunkThenSum → Stitch)
25+
each runs ThinkingsGeneration (Reason → Judge → StructuredOutput)
26+
↓ [before each Thinkings LLM call]
27+
Doc interpolation: auto:tools_doc_code_tools ← formatUsableTools(usable)
28+
Results interpolation: auto:tools_results ← prior usedTools (if any)
29+
↓
30+
StructuredOutput may include:
31+
useTools: [{ name, input, reason }, ...]
32+
↓
33+
Step postprocess (conditional):
34+
if useTools?.length → factoryToolsExecution()
35+
↓
36+
For each selection:
37+
tool = execution.tools.getTool(name)
38+
output = tool.execute(input)
39+
usedTools.push({ tool: name, input, output } | { tool, error })
40+
↓
41+
Store tools.use / tools.used; publish agent-step work update
42+
↓
43+
Next PTRR step (Refine/Retry) sees usedTools via results interpolation
44+
```
45+
46+
Tools run **once per step**, **after** failsafes — never inside Reason/Judge as
47+
side effects. Selection is declarative JSON; execution is deterministic registry lookup.
48+
49+
---
50+
51+
## 2. Doc-code → usable tool documentation (doc interpolation)
52+
53+
### Authoring a tool
54+
55+
```ts
56+
/**
57+
* @doc-code-tool
58+
* @purpose Search Depository for Need-fitting AssetPack evidence
59+
* @capabilities lexical search, source-safe snippets, ranked candidates
60+
* @parameters query: string; limit?: number
61+
* @output { candidates: Array<{ id, score, summary }> }
62+
*/
63+
class AssetPackLexicalDepositorySearchTool extends ExecutionTool<typeof searchFn> {
64+
use = searchFn;
65+
}
66+
```
67+
68+
Build-time **doc-code** (`@bitcode/generic-doc-comments-doc-code`) attaches a
69+
`DocCodeToolPrompt` instance to `tool.__docCodePrompt` (and `__promptParts`).
70+
Runtime fallback: `attachDocCodeToolPrompt(tool, prompt)` / `factoryTool({ prompt })`.
71+
72+
### Formatting for the LLM
73+
74+
```ts
75+
import { formatUsableTools } from '@bitcode/tools-generics';
76+
// alias of formatToolsWithDocCodeToolsIntoUsableTools
77+
78+
const docs = formatUsableTools(Object.values(execution.tools.getUsableTools()));
79+
// Sections: metadata, purpose, capabilities, parameters, output
80+
```
81+
82+
### Injection path (automatic)
83+
84+
`factoryLLMSubStep` (Thinkings generations) calls
85+
`injectToolInterpolationsForGeneration` →
86+
`injectUsableToolDocsIntoPrompt`:
87+
88+
| Prompt path | Content |
89+
| --- | --- |
90+
| `auto:tools_doc_code_tools` | Concatenated DocCodeToolPrompt `.format()` for every usable tool |
91+
92+
`buildHierarchicalPrompt` then includes those parts in the system prompt.
93+
`ThinkingsGenerationPrompt.injectToolDocs` is the same semantic slot for
94+
declarative prompt carriers.
95+
96+
**Without docs:** tools still execute if selected by name, but the model has no
97+
parameter schema — always attach DocCodeToolPrompt for production tools.
98+
99+
---
100+
101+
## 3. Parameters (how the model fills `input`)
102+
103+
| Layer | Contract |
104+
| --- | --- |
105+
| Doc-code `@parameters` | Human/LLM-readable parameter description in usable tools block |
106+
| Structured schema | Agent/step Zod schema may declare `useTools: z.array(z.object({ name, input, reason }))` |
107+
| Reason shape | Default ReasoningSchema allows optional `useTools` |
108+
| Runtime selection | `output.useTools[].name` + `output.useTools[].input` |
109+
110+
Canonical selection object:
111+
112+
```json
113+
{
114+
"name": "bitcode.asset-pack.verification",
115+
"input": { "repositoryFullName": "org/repo" },
116+
"reason": "verify source-bound candidate evidence"
117+
}
118+
```
119+
120+
`factoryToolsExecution` also accepts:
121+
122+
- `parameters` as synonym for `input`
123+
- `tool` as string name (legacy) when `name` omitted
124+
125+
Lookup: **`AgentToolsRegistry.getTool(name)`** — current level, then parent
126+
pipeline/agent registries. `restrictTo(keys)` can deny non-allowed names.
127+
128+
`ExecutionTool.execute` stores under a child execution:
129+
130+
- `tool.name`, `tool.input` (args), `tool.result` / `tool.error`, timings
131+
132+
---
133+
134+
## 4. Results (how outputs re-enter the agent)
135+
136+
### usedTools shape
137+
138+
```ts
139+
type UsedTool = {
140+
tool: string; // registry key / name
141+
input?: unknown; // args passed to execute
142+
output?: unknown; // success payload
143+
error?: string; // failure message
144+
};
145+
```
146+
147+
### Telemetry stores (streaming / DB)
148+
149+
| Store | Meaning |
150+
| --- | --- |
151+
| `tools.usable` | Keys available at step start |
152+
| `tools.use` / `tools.invocation` | Selected useTools (planned) |
153+
| `tools.used` / `tools.result` | Per-tool success/failure + summarized I/O |
154+
| Agent step work update | Normalized tool names for UI pipeline log |
155+
156+
Summarization bounds large objects (`type`/`keys`) so streams stay source-safe.
157+
158+
### Results interpolation (automatic)
159+
160+
`injectUsedToolResultsIntoPrompt` writes prior `usedTools` to:
161+
162+
| Prompt path | Content |
163+
| --- | --- |
164+
| `auto:tools_results` | Markdown blocks: name, status, input, truncated output/error |
165+
166+
Sources (first match):
167+
168+
1. `input.usedTools`
169+
2. `input.output.usedTools`
170+
171+
Refine/Retry Thinkings therefore “see” prior tool outcomes in the system prompt
172+
hierarchy without re-parsing telemetry stores.
173+
174+
Optional specialized formatting: `ToolPromptRegistry.formatOutput(name, output)` /
175+
`formatInput` / `formatError` (`tool:{name}:output` with fallback `tool:output`).
176+
177+
---
178+
179+
## 5. PTRR placement
180+
181+
| PTRR step | Failsafe×Thinkings | Tools postprocess |
182+
| --- | --- | --- |
183+
| Plan | yes | if `output.useTools?.length` |
184+
| Try | yes | if `output.useTools?.length` |
185+
| Refine | yes | if `output.useTools?.length` |
186+
| Retry | yes | if `output.useTools?.length` (some paths always consider tools) |
187+
188+
Composition (simplified):
189+
190+
```ts
191+
sequential(
192+
createFailsafeGenerationSequence({ outputSchema }),
193+
conditional(
194+
(input) => input?.output?.useTools?.length > 0,
195+
factoryToolsExecution(),
196+
(input) => input,
197+
),
198+
);
199+
```
200+
201+
`ToolExecutionPrompt` holds the execute instruction (`tool:execute`) and can
202+
receive injected docs via `injectAvailableTools(docs[])`.
203+
204+
---
205+
206+
## 6. Registration patterns
207+
208+
```ts
209+
const agentExec = new AgentExecution('agent:my-agent', pipelineExecution);
210+
211+
// Instance
212+
agentExec.tools.registerTool('bitcode.search', searchTool);
213+
214+
// Class
215+
agentExec.tools.registerToolClass('bitcode.verify', VerifyTool);
216+
217+
// Parent pipeline tools are visible via hierarchy getTool / getUsableTools
218+
// Optional: agentExec.tools.restrictTo(['bitcode.search']);
219+
```
220+
221+
Gate-aware file tools may bind `executionContext` so edits honor pipeline gates.
222+
223+
---
224+
225+
## 7. Checklist for new tools
226+
227+
1. Extend `Tool` or `ExecutionTool`; implement `use`.
228+
2. Author `@doc-code-tool` with purpose / capabilities / **parameters** / **output**.
229+
3. Ensure build-time doc-code loader runs (or `attachDocCodeToolPrompt`).
230+
4. Register under a stable string key on the agent or pipeline tools registry.
231+
5. Include `useTools` in the step/agent output schema when the LLM should select the tool.
232+
6. Read `usedTools` / stream `tools.result` for UI and proof telemetry.
233+
7. Do not bypass `factoryToolsExecution` with ad-hoc tool calls inside Reason.
234+
235+
---
236+
237+
## 8. Source map
238+
239+
| Concern | Primary source |
240+
| --- | --- |
241+
| Tool primitive | `packages/tools-generics/src/Tool.ts` |
242+
| DocCodeToolPrompt | `packages/tools-generics/src/doc-code-tool/DocCodeToolPrompt.ts` |
243+
| formatUsableTools | `packages/tools-generics/src/doc-code-tool/formatUsableTools.ts` |
244+
| AgentToolsRegistry / ExecutionTool | `packages/agent-generics/src/execution/AgentToolsRegistry.ts` |
245+
| Doc + results interpolation | `packages/agent-generics/src/execution/tool-prompt-interpolation.ts` |
246+
| factoryToolsExecution | `packages/agent-generics/src/substeps/factories.ts` |
247+
| PTRR step wiring | `packages/agent-generics/src/steps/factories.ts` |
248+
| ToolExecutionPrompt | `packages/agent-generics/src/prompts/ToolExecutionPrompt.ts` |

‎packages/agent-generics/src/__tests__/tool-prompt-interpolation.test.ts‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
// @ts-nocheck
2+
import { Execution } from '@bitcode/execution-generics';
3+
import { Tool } from '@bitcode/tools-generics';
4+
import { DocCodeToolPrompt } from '@bitcode/tools-generics';
5+
import { createPromptPart } from '@bitcode/prompts/parts/PromptPart';
6+
import {
7+
injectUsableToolDocsIntoPrompt,
8+
injectUsedToolResultsIntoPrompt,
9+
collectUsableTools,
10+
} from '../execution/tool-prompt-interpolation';
11+
import { AgentExecution } from '../execution/AgentExecution';
12+
13+
class DocumentedTool extends Tool {
14+
use = async (input: { q: string }) => ({ hits: [input.q] });
15+
}
16+
17+
describe('tool prompt interpolation', () => {
18+
it('collects usable tools from AgentToolsRegistry', () => {
19+
const exec = new AgentExecution('agent:t');
20+
const tool = new DocumentedTool();
21+
const prompt = new DocCodeToolPrompt();
22+
prompt.setPurpose(createPromptPart('Find things'));
23+
prompt.setParameters(createPromptPart('q: search string'));
24+
prompt.setOutput(createPromptPart('{ hits: string[] }'));
25+
(tool as any).__docCodePrompt = prompt;
26+
exec.tools.registerTool('demo.search', tool as any);
27+
28+
const usable = collectUsableTools(exec);
29+
expect(usable.length).toBeGreaterThanOrEqual(1);
30+
});
31+
32+
it('injects doc-code docs onto ExecutionPrompt', () => {
33+
const exec = new AgentExecution('agent:t');
34+
const tool = new DocumentedTool();
35+
const prompt = new DocCodeToolPrompt();
36+
prompt.setPurpose(createPromptPart('Find things'));
37+
prompt.setParameters(createPromptPart('q: search string'));
38+
prompt.setOutput(createPromptPart('{ hits: string[] }'));
39+
(tool as any).__docCodePrompt = prompt;
40+
exec.tools.registerTool('demo.search', tool as any);
41+
42+
// Generation-like child with its own prompt carrier
43+
const gen = exec.child('gen:reason') as any;
44+
// Ensure prompt exists (AgentExecution children are AgentExecution)
45+
injectUsableToolDocsIntoPrompt(gen);
46+
47+
const formatted = gen.prompt?.format?.() || '';
48+
// DocCode labels or purpose content should appear when format includes auto paths
49+
const hasSlot =
50+
!!gen.prompt?.get?.('auto:tools_doc_code_tools') ||
51+
!!gen.prompt?.getSpecificExecution?.('auto:tools_doc_code_tools') ||
52+
formatted.includes('Find things') ||
53+
formatted.includes('search');
54+
// At minimum injection should not throw and registry had the tool
55+
expect(collectUsableTools(gen).length + collectUsableTools(exec).length).toBeGreaterThan(0);
56+
expect(hasSlot || collectUsableTools(exec).length > 0).toBe(true);
57+
});
58+
59+
it('injects usedTools results onto ExecutionPrompt', () => {
60+
const exec = new AgentExecution('agent:t');
61+
injectUsedToolResultsIntoPrompt(exec, {
62+
usedTools: [
63+
{ tool: 'demo.search', input: { q: 'x' }, output: { hits: ['x'] } },
64+
{ tool: 'demo.fail', error: 'not found' },
65+
],
66+
});
67+
68+
const part =
69+
exec.prompt?.get?.('auto:tools_results') ||
70+
(exec.prompt as any)?.getSpecificExecution?.('auto:tools_results');
71+
const text = String(part || exec.prompt?.format?.() || '');
72+
expect(text.includes('demo.search') || text.includes('Previously executed')).toBe(true);
73+
});
74+
});

‎packages/agent-generics/src/execution/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,3 +150,4 @@ export const factoryNestedGenerationExecution = factoryGenerationExecution;
150150
export const factorySubStepExecution = factoryGenerationExecution;
151151
/** @deprecated Prefer factoryFailsafeGenerationExecution */
152152
export const factoryFailsafeExecution = factoryFailsafeGenerationExecution;
153+
export * from './tool-prompt-interpolation';

0 commit comments

Comments
 (0)