Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions docs/evidence/pr-29-04/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# PR 29: live classification on Poppy 0.4

This refresh targets main at `b2fc42893ce719023681ff0df14a2b485e6d75d3`. The tested implementation and capture source is `3f240b5f43c97a6012b612f1d0f4e2e0cb7e15e9`.

The runner changes classify missing live capabilities and statically required fixture values as unsupported. Local adapters that claim a scenario still fail. Fixture preflight shares execution's two interpolation passes, checks optional parameters and fixture-known conditions, and defers unknown runtime expressions. The current currency and feedback concurrency tests from main are retained without changes.

The [previous PR snapshot](https://github.com/adewale/pda/tree/d02855ec490ee0076ecc9a1549dba30004b012a3) preserves the October reports, 0.3 screenshots and their verification records. Those records are historical and are not carried into this focused change.

## Verification

- [Harness](interop.txt): 474 tests passed in 34 files.
- [Negative control](preflight-disabled.txt): all four CLI regressions fail when fixture preflight is disabled. The source was restored before the full suite and capture.
- [Full suite](tests-summary.txt): **2,705 passed, one existing skip**, across 134 files.
- Type checking, pinned-spec integrity, documentation links and all 17 tooling tests passed.
- [Validation receipt](validation.json) records source and artifact hashes.

## Visual evidence

Reviewer cue: `fixture-live` changes from red **fail** to grey **n/a** for both `CL` and `ML`. `fixture-claimed` stays red **fail** in both scenarios.

| Before: absent live capabilities fail | After: absent live capabilities are unsupported |
| --- | --- |
| ![Base 0.4 report: both fixture columns fail CL and ML](before.png) | ![Updated 0.4 report: live capabilities are unsupported, while claimed-support controls still fail](after.png) |

The same synthetic discovery response runs through actual immutable `runCell` and scenario sources. The production renderer and its complete runtime dependency closure are loaded from each revision, checked for equality and hashed. Chromium renders both at 1400 × 1000; external page requests are blocked. Assertions check all four outcomes, that only discovery runs, and the rendered grid counts. [Receipt](visual-receipt.json), [base HTML](before.html), [updated HTML](after.html).

Reproduce with the pinned dependencies, Node 24 and Chromium:

```sh
node docs/evidence/pr-29-04/capture.mjs \
b2fc42893ce719023681ff0df14a2b485e6d75d3 \
3f240b5f43c97a6012b612f1d0f4e2e0cb7e15e9
```

These are synthetic classification checks. They do not establish live Company interoperability or an authenticated hosted journey. No public Company was contacted for this refresh, and the registry's permitted live scenarios are unchanged.
85 changes: 85 additions & 0 deletions docs/evidence/pr-29-04/after.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
<!doctype html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="width=device-width, initial-scale=1">
<title>Poppy interop PR29 synthetic capability fixture</title>
<style>
:root { color-scheme: light dark; --pass: #1a7f37; --fail: #cf222e; --na: #6e7781; --error: #9a6700; }
body { font: 15px/1.45 system-ui, sans-serif; max-width: 72rem; margin: 0 auto; padding: 1rem; }
table { border-collapse: collapse; margin: 0.5rem 0 1rem; }
th, td { border: 1px solid #8884; padding: 0.25rem 0.5rem; text-align: left; vertical-align: top; }
.grid td { min-width: 9rem; }
td.pass a, span.pass { color: var(--pass); }
td.fail a, span.fail, tr.bad td, tr.error td:first-child { color: var(--fail); }
td.unsupported a, span.unsupported, td.none { color: var(--na); }
td.error a, span.error { color: var(--error); }
td.known { text-decoration: underline dotted; }
.meta { color: var(--na); margin-top: -0.5rem; }
details { border-top: 1px solid #8884; padding: 0.5rem 0; }
code { font-size: 0.9em; }
</style>
<h1>Poppy interop run PR29 synthetic capability fixture</h1>
<p>Lane synthetic; no external traffic · Poppy 0.4 · 2 unsupported, 2 fail.
Checks: 0 failures, 0 warnings, 0 tentative (our readings); only failures fail a cell (docs/conformance-levels.md).
Each failed cell is blamed by the first rule that decides it: violation, warning, grid, reading, or unattributed (docs/interop-spec.md, section 9).</p>


<h2>Implementations</h2>
<table><thead><tr><th>name</th><th>role</th><th>commit</th><th>readings</th><th>interop-only relaxations</th></tr></thead><tbody><tr><td>pda</td><td>agent</td><td><code>3f240b5f43c97a6012b612f1d0f4e2e0cb7e15e9</code></td><td></td><td>Synthetic discover response; actual runCell/report code</td></tr>
<tr><td>fixture-live</td><td>company</td><td><code>fixture</code></td><td></td><td>No conversations or MCP</td></tr>
<tr><td>fixture-claimed</td><td>company</td><td><code>fixture</code></td><td></td><td>No conversations or MCP</td></tr></tbody></table>
<section>
<h2 id="CL">CL: Conversation at a live Company: a labelled test message, read, a follow-up, then close</h2>
<p class=meta>Spec 8.2, 8.3, 8.5, 8.12</p>
<table class=grid><thead><tr><th>agent \ Company</th><th scope=col>fixture-live</th><th scope=col>fixture-claimed</th></tr></thead>
<tbody><tr><th scope=row>pda</th><td class="unsupported"><a href="#CL--pda--fixture-live">n/a</a></td><td class="fail"><a href="#CL--pda--fixture-claimed">fail</a></td></tr></tbody></table>
</section>
<section>
<h2 id="ML">ML: MCP tool listing without calling a tool</h2>
<p class=meta>Spec 7</p>
<table class=grid><thead><tr><th>agent \ Company</th><th scope=col>fixture-live</th><th scope=col>fixture-claimed</th></tr></thead>
<tbody><tr><th scope=row>pda</th><td class="unsupported"><a href="#ML--pda--fixture-live">n/a</a></td><td class="fail"><a href="#ML--pda--fixture-claimed">fail</a></td></tr></tbody></table>
</section>

<h2>Cells</h2>
<details id="CL--pda--fixture-live">
<summary><b>CL</b> pda × fixture-live: <span class="unsupported">n/a</span></summary>
<p>step 1 (expect): the Company doesn&#39;t offer what CL tests: ${d.conversations}: expected true, got false</p>



<table><thead><tr><th>#</th><th>actor</th><th>op</th><th>ms</th><th>result</th></tr></thead><tbody><tr class="ok"><td>0</td><td>agent</td><td>discover</td><td>0</td><td>ok</td></tr>
<tr class="bad"><td>1</td><td>expect</td><td>expect</td><td>0</td><td>${d.conversations}: expected true, got false</td></tr></tbody></table>
<p>No checks reported anything.</p>
</details>
<details id="CL--pda--fixture-claimed" open>
<summary><b>CL</b> pda × fixture-claimed: <span class="fail">fail</span></summary>
<p>step 1 (expect): ${d.conversations}: expected true, got false</p>



<table><thead><tr><th>#</th><th>actor</th><th>op</th><th>ms</th><th>result</th></tr></thead><tbody><tr class="ok"><td>0</td><td>agent</td><td>discover</td><td>0</td><td>ok</td></tr>
<tr class="bad"><td>1</td><td>expect</td><td>expect</td><td>0</td><td>${d.conversations}: expected true, got false</td></tr></tbody></table>
<p>No checks reported anything.</p>
</details>
<details id="ML--pda--fixture-live">
<summary><b>ML</b> pda × fixture-live: <span class="unsupported">n/a</span></summary>
<p>step 1 (expect): the Company doesn&#39;t offer what ML tests: ${d.apis}: expected $contains &quot;mcp&quot;, got []</p>



<table><thead><tr><th>#</th><th>actor</th><th>op</th><th>ms</th><th>result</th></tr></thead><tbody><tr class="ok"><td>0</td><td>agent</td><td>discover</td><td>0</td><td>ok</td></tr>
<tr class="bad"><td>1</td><td>expect</td><td>expect</td><td>0</td><td>${d.apis}: expected $contains &quot;mcp&quot;, got []</td></tr></tbody></table>
<p>No checks reported anything.</p>
</details>
<details id="ML--pda--fixture-claimed" open>
<summary><b>ML</b> pda × fixture-claimed: <span class="fail">fail</span></summary>
<p>step 1 (expect): ${d.apis}: expected $contains &quot;mcp&quot;, got []</p>



<table><thead><tr><th>#</th><th>actor</th><th>op</th><th>ms</th><th>result</th></tr></thead><tbody><tr class="ok"><td>0</td><td>agent</td><td>discover</td><td>0</td><td>ok</td></tr>
<tr class="bad"><td>1</td><td>expect</td><td>expect</td><td>0</td><td>${d.apis}: expected $contains &quot;mcp&quot;, got []</td></tr></tbody></table>
<p>No checks reported anything.</p>
</details>
</html>
Binary file added docs/evidence/pr-29-04/after.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
85 changes: 85 additions & 0 deletions docs/evidence/pr-29-04/before.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
<!doctype html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="width=device-width, initial-scale=1">
<title>Poppy interop PR29 synthetic capability fixture</title>
<style>
:root { color-scheme: light dark; --pass: #1a7f37; --fail: #cf222e; --na: #6e7781; --error: #9a6700; }
body { font: 15px/1.45 system-ui, sans-serif; max-width: 72rem; margin: 0 auto; padding: 1rem; }
table { border-collapse: collapse; margin: 0.5rem 0 1rem; }
th, td { border: 1px solid #8884; padding: 0.25rem 0.5rem; text-align: left; vertical-align: top; }
.grid td { min-width: 9rem; }
td.pass a, span.pass { color: var(--pass); }
td.fail a, span.fail, tr.bad td, tr.error td:first-child { color: var(--fail); }
td.unsupported a, span.unsupported, td.none { color: var(--na); }
td.error a, span.error { color: var(--error); }
td.known { text-decoration: underline dotted; }
.meta { color: var(--na); margin-top: -0.5rem; }
details { border-top: 1px solid #8884; padding: 0.5rem 0; }
code { font-size: 0.9em; }
</style>
<h1>Poppy interop run PR29 synthetic capability fixture</h1>
<p>Lane synthetic; no external traffic · Poppy 0.4 · 4 fail.
Checks: 0 failures, 0 warnings, 0 tentative (our readings); only failures fail a cell (docs/conformance-levels.md).
Each failed cell is blamed by the first rule that decides it: violation, warning, grid, reading, or unattributed (docs/interop-spec.md, section 9).</p>


<h2>Implementations</h2>
<table><thead><tr><th>name</th><th>role</th><th>commit</th><th>readings</th><th>interop-only relaxations</th></tr></thead><tbody><tr><td>pda</td><td>agent</td><td><code>b2fc42893ce719023681ff0df14a2b485e6d75d3</code></td><td></td><td>Synthetic discover response; actual runCell/report code</td></tr>
<tr><td>fixture-live</td><td>company</td><td><code>fixture</code></td><td></td><td>No conversations or MCP</td></tr>
<tr><td>fixture-claimed</td><td>company</td><td><code>fixture</code></td><td></td><td>No conversations or MCP</td></tr></tbody></table>
<section>
<h2 id="CL">CL: Conversation at a live Company: a labelled test message, read, a follow-up, then close</h2>
<p class=meta>Spec 8.2, 8.3, 8.5, 8.12</p>
<table class=grid><thead><tr><th>agent \ Company</th><th scope=col>fixture-live</th><th scope=col>fixture-claimed</th></tr></thead>
<tbody><tr><th scope=row>pda</th><td class="fail"><a href="#CL--pda--fixture-live">fail</a></td><td class="fail"><a href="#CL--pda--fixture-claimed">fail</a></td></tr></tbody></table>
</section>
<section>
<h2 id="ML">ML: MCP tool listing without calling a tool</h2>
<p class=meta>Spec 7</p>
<table class=grid><thead><tr><th>agent \ Company</th><th scope=col>fixture-live</th><th scope=col>fixture-claimed</th></tr></thead>
<tbody><tr><th scope=row>pda</th><td class="fail"><a href="#ML--pda--fixture-live">fail</a></td><td class="fail"><a href="#ML--pda--fixture-claimed">fail</a></td></tr></tbody></table>
</section>

<h2>Cells</h2>
<details id="CL--pda--fixture-live" open>
<summary><b>CL</b> pda × fixture-live: <span class="fail">fail</span></summary>
<p>step 1 (expect): ${d.conversations}: expected true, got false</p>



<table><thead><tr><th>#</th><th>actor</th><th>op</th><th>ms</th><th>result</th></tr></thead><tbody><tr class="ok"><td>0</td><td>agent</td><td>discover</td><td>0</td><td>ok</td></tr>
<tr class="bad"><td>1</td><td>expect</td><td>expect</td><td>1</td><td>${d.conversations}: expected true, got false</td></tr></tbody></table>
<p>No checks reported anything.</p>
</details>
<details id="CL--pda--fixture-claimed" open>
<summary><b>CL</b> pda × fixture-claimed: <span class="fail">fail</span></summary>
<p>step 1 (expect): ${d.conversations}: expected true, got false</p>



<table><thead><tr><th>#</th><th>actor</th><th>op</th><th>ms</th><th>result</th></tr></thead><tbody><tr class="ok"><td>0</td><td>agent</td><td>discover</td><td>0</td><td>ok</td></tr>
<tr class="bad"><td>1</td><td>expect</td><td>expect</td><td>0</td><td>${d.conversations}: expected true, got false</td></tr></tbody></table>
<p>No checks reported anything.</p>
</details>
<details id="ML--pda--fixture-live" open>
<summary><b>ML</b> pda × fixture-live: <span class="fail">fail</span></summary>
<p>step 1 (expect): ${d.apis}: expected $contains &quot;mcp&quot;, got []</p>



<table><thead><tr><th>#</th><th>actor</th><th>op</th><th>ms</th><th>result</th></tr></thead><tbody><tr class="ok"><td>0</td><td>agent</td><td>discover</td><td>0</td><td>ok</td></tr>
<tr class="bad"><td>1</td><td>expect</td><td>expect</td><td>0</td><td>${d.apis}: expected $contains &quot;mcp&quot;, got []</td></tr></tbody></table>
<p>No checks reported anything.</p>
</details>
<details id="ML--pda--fixture-claimed" open>
<summary><b>ML</b> pda × fixture-claimed: <span class="fail">fail</span></summary>
<p>step 1 (expect): ${d.apis}: expected $contains &quot;mcp&quot;, got []</p>



<table><thead><tr><th>#</th><th>actor</th><th>op</th><th>ms</th><th>result</th></tr></thead><tbody><tr class="ok"><td>0</td><td>agent</td><td>discover</td><td>0</td><td>ok</td></tr>
<tr class="bad"><td>1</td><td>expect</td><td>expect</td><td>0</td><td>${d.apis}: expected $contains &quot;mcp&quot;, got []</td></tr></tbody></table>
<p>No checks reported anything.</p>
</details>
</html>
Binary file added docs/evidence/pr-29-04/before.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
95 changes: 95 additions & 0 deletions docs/evidence/pr-29-04/capture.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
/** One-off PR evidence: use immutable runCell sources and the unchanged production report renderer. */
import assert from "node:assert/strict";
import { execFileSync, spawn } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { pathToFileURL } from "node:url";
import puppeteer from "@cloudflare/puppeteer";

const [base, head] = process.argv.slice(2);
assert.ok([base, head].every(sha => /^[a-f0-9]{40}$/.test(sha)), "Supply full base/head SHAs");
const root = path.resolve(import.meta.dirname, "../../..");
const output = import.meta.dirname;
const directory = mkdtempSync(path.join(tmpdir(), "pr29-04-classification-"));
const git = (sha, file) => execFileSync("git", ["show", `${sha}:${file}`], { cwd: root });
const hash = bytes => createHash("sha256").update(bytes).digest("hex");
const rendererSources = [
"interop/runner/report.ts", "interop/runner/attribution.ts", "interop/runner/readings.ts", "interop/runner/har.ts",
"src/validator/levels.ts", "src/poppy/protocol-version.ts"
];
for (const file of rendererSources) assert.deepEqual(git(base, file), git(head, file), `Renderer dependency must be identical: ${file}`);
const receipt = {
base, head, fixture: "Company offers neither conversations nor MCP", networkRequests: 0,
rendererSha256: Object.fromEntries(rendererSources.map(file => [file, hash(git(head, file))])), stages: []
};
let child, browser;
try {
child = spawn(process.env.PDA_ACCEPTANCE_CHROMIUM || "/usr/bin/chromium", ["--headless", "--no-sandbox", "--disable-dev-shm-usage", "--disable-background-networking", "--disable-component-update", "--no-first-run", "--remote-debugging-port=0", `--user-data-dir=${path.join(directory, "chrome")}`], { stdio: ["ignore", "ignore", "pipe"] });
const endpoint = await new Promise((resolve, reject) => {
let stderr = "";
const timer = setTimeout(() => reject(new Error("Chromium startup timed out")), 10_000);
child.once("error", error => { clearTimeout(timer); reject(error); });
child.stderr.on("data", chunk => {
stderr = (stderr + String(chunk)).slice(-8192);
const match = /DevTools listening on (ws:\/\/127\.0\.0\.1:[0-9]+\/devtools\/browser\/[a-z0-9-]+)/.exec(stderr);
if (match) { clearTimeout(timer); resolve(match[1]); }
});
});
browser = await puppeteer.connect({ browserWSEndpoint: endpoint });
const page = await browser.newPage();
await page.setViewport({ width: 1400, height: 1000, deviceScaleFactor: 1 });
await page.setRequestInterception(true);
page.on("request", request => { void request.abort(); });
for (const [stage, sha] of [["before", base], ["after", head]]) {
const source = path.join(directory, stage);
for (const file of ["interop/runner/cell.ts", "interop/runner/scenario.ts", ...rendererSources]) {
const target = path.join(source, file);
mkdirSync(path.dirname(target), { recursive: true });
writeFileSync(target, git(sha, file));
}
const { runCell } = await import(pathToFileURL(path.join(source, "interop/runner/cell.ts")));
const { renderReport } = await import(pathToFileURL(path.join(source, "interop/runner/report.ts")));
const scenarios = ["CL", "ML"].map(id => JSON.parse(git(sha, `interop/scenarios/${id}.json`)));
const cells = [];
for (const scenario of scenarios) for (const live of [true, false]) {
const calls = [];
const driver = {
agent: async op => {
calls.push(op);
assert.equal(op, "discover", "An absent feature must stop before sending or listing");
return { kind: "ok", result: { poppy_version: "0.4", conversations: false, apis: [] } };
},
company: async () => { throw new Error("No Company calls in this fixture"); },
runner: async () => { throw new Error("No runner calls in this fixture"); }
};
const run = await runCell(scenario, {}, driver, { live });
assert.deepEqual(calls, ["discover"]);
assert.equal(run.outcome, stage === "after" && live ? "unsupported" : "fail");
cells.push({ agent: "pda", company: live ? "fixture-live" : "fixture-claimed", scenario: scenario.id, protocolVersion: "0.4", outcome: run.outcome, reason: run.reason, blame: null, rule: null, steps: run.steps, checks: [], har: null });
}
const html = renderReport({
run: "PR29 synthetic capability fixture", lane: "synthetic; no external traffic", spec: "Poppy 0.4",
implementations: [
{ name: "pda", role: "agent", commit: sha, readings: {}, relaxations: ["Synthetic discover response; actual runCell/report code"] },
...["fixture-live", "fixture-claimed"].map(name => ({ name, role: "company", commit: "fixture", readings: {}, relaxations: ["No conversations or MCP"] }))
], scenarios, cells
});
const htmlFile = path.join(output, `${stage}.html`);
const pngFile = path.join(output, `${stage}.png`);
writeFileSync(htmlFile, html);
await page.setContent(html, { waitUntil: "load", timeout: 10_000 });
await page.waitForSelector(".grid");
assert.equal(await page.$$eval(".grid td.fail", cells => cells.length), stage === "before" ? 4 : 2);
assert.equal(await page.$$eval(".grid td.unsupported", cells => cells.length), stage === "before" ? 0 : 2);
await page.evaluate(() => { window.scrollTo(0, 0); });
writeFileSync(pngFile, await page.screenshot({ type: "png" }));
receipt.stages.push({ stage, source: sha, cells: cells.map(({ company, scenario, outcome }) => ({ company, scenario, outcome })), htmlSha256: hash(Buffer.from(html)) });
}
writeFileSync(path.join(output, "visual-receipt.json"), `${JSON.stringify(receipt, null, 2)}\n`);
} finally {
await browser?.close().catch(() => undefined);
child?.kill("SIGTERM");
rmSync(directory, { recursive: true, force: true });
}
7 changes: 7 additions & 0 deletions docs/evidence/pr-29-04/interop.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
RUN v4.1.12 /workspace/pda


Test Files 34 passed (34)
Tests 474 passed (474)
Start at 23:30:34
Duration 6.44s (transform 2.36s, setup 0ms, import 5.71s, tests 6.58s, environment 4ms)
Loading