diff --git a/README.md b/README.md index 5974158..47195d2 100644 --- a/README.md +++ b/README.md @@ -221,6 +221,32 @@ close, browser sessions, feedback, and Operations. What remains: Each open or partial requirement is listed with its reason in [Protocol coverage](https://personal-agent-protocol.datalayer.tech/protocol/coverage/). +## Conformance and independent testing + +- **The SDKs' own report**: every release runs each positive, negative and + recovery fixture across the four pairings (a Python or TypeScript personal + agent against a Python or TypeScript reference company), over real TCP and + TLS: [Compatibility and conformance report](https://personal-agent-protocol.datalayer.tech/protocol/conformance-report/). +- **An independent study**: in November 2026 an independent tester ran the + Python SDK 0.35.0, self-hosted and unmodified, through Poppy 1.0's Operations + extension and sign-out tests, with effects counted by a separate recorder. + Everything passed: a retry after a lost response returned the original + response and performed the action once (3 of 3 trials; a deliberately broken + store was caught performing it twice); after sign-out, refresh was refused + with `invalid_grant` and an earlier access token was refused with no effect; + and the agent side passed six rules (declines, revised terms after an agent + approval, lost responses, `Retry-After`, company text never counting as the + user's approval, instruction coverage). One finding is about the protocol, not + the SDK: an operation already performing at sign-out completed and reported + success, which Poppy leaves unspecified (raised on the Poppy tracker, #84). + 0.36.x was not tested. The write-up, evidence and harness: + [studies/poppy-1.0](https://github.com/kkrlstrm/kirkaldy-conformance-kit/tree/main/studies/poppy-1.0). +- **The official company kit**: validate a company implementation, on these + SDKs or any other, with the protocol's + [kit/company](https://github.com/personal-agent-protocol/personal-agent-protocol/tree/main/kit/company). + +See [Independent testing and the official kit](https://personal-agent-protocol.datalayer.tech/protocol/independent-testing/). + ## Security boundary PAP credentials belong to the host, not to an agent model or tool call. diff --git a/docs/docs/protocol/independent-testing.mdx b/docs/docs/protocol/independent-testing.mdx new file mode 100644 index 0000000..276422a --- /dev/null +++ b/docs/docs/protocol/independent-testing.mdx @@ -0,0 +1,50 @@ +--- +title: Independent testing and the official kit +sidebar_position: 5 +--- + +# Independent testing and the official kit + +Besides the SDKs' own [conformance report](./conformance-report.mdx), two outside +checks apply to these SDKs: an independent study of the Python SDK against +Poppy 1.0, and the protocol's official company kit, which anyone can run against +their own implementation. + +## An independent study of the Python SDK 0.35.0 + +In November 2026 an independent tester of agent infrastructure ran the +**Python SDK 0.35.0, self-hosted and unmodified**, through tests of Poppy 1.0's +**Operations extension** and **sign-out**. The effects of each test were counted +by a separate recorder, not read from the SDK's own responses. + +**Everything passed:** + +| Area | What was tested | Result | +| --- | --- | --- | +| Retry after a lost response | The company answers again with the original response and performs the action once | Passed in 3 of 3 trials. A deliberately broken store performed the action twice, and the test caught it. | +| Sign-out | After sign-out, refresh is refused with `invalid_grant`, and an access token issued before sign-out is refused on the next request, with no effect | Passed | +| The agent side, six rules | Declines; revised terms after an agent's approval; lost responses; `Retry-After`; company text never counting as the user's approval; instruction coverage | Passed | + +**One observation about the protocol, not the SDK:** an operation that was +already performing when the user signed out completed and reported success. +Poppy does not say what should happen in that case; it was raised on the Poppy +tracker (issue #84). + +**What it does not cover:** the study tested 0.35.0. Version 0.36.x (the 1.0 +draft as the default) has not been tested by it. + +The write-up, its evidence and the harness are published: +[studies/poppy-1.0](https://github.com/kkrlstrm/kirkaldy-conformance-kit/tree/main/studies/poppy-1.0). +The same study proposes Operations checks for the official validator, which does +not test Operations yet, citing the lost-response test on this SDK as its +measured example. + +## The official company kit + +The Personal Agent Protocol project publishes a kit to validate a company's +implementation: +[kit/company](https://github.com/personal-agent-protocol/personal-agent-protocol/tree/main/kit/company) +in the protocol's repository. Run it against a company built on these SDKs, or on +any other implementation, before publishing its `poppy.json`. It is the +reference check of a company; this site's [conformance report](./conformance-report.mdx) +is the SDKs' own, run on every release.