Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,32 @@ close, browser sessions, feedback, and Operations. What remains:
Each open or partial requirement is listed with its reason in
[Protocol coverage](https://personal-agent-protocol.datalayer.tech/protocol/coverage/).

## Conformance and independent testing

- **The SDKs' own report**: every release runs each positive, negative and
recovery fixture across the four pairings (a Python or TypeScript personal
agent against a Python or TypeScript reference company), over real TCP and
TLS: [Compatibility and conformance report](https://personal-agent-protocol.datalayer.tech/protocol/conformance-report/).
- **An independent study**: in November 2026 an independent tester ran the
Python SDK 0.35.0, self-hosted and unmodified, through Poppy 1.0's Operations
extension and sign-out tests, with effects counted by a separate recorder.
Everything passed: a retry after a lost response returned the original
response and performed the action once (3 of 3 trials; a deliberately broken
store was caught performing it twice); after sign-out, refresh was refused
with `invalid_grant` and an earlier access token was refused with no effect;
and the agent side passed six rules (declines, revised terms after an agent
approval, lost responses, `Retry-After`, company text never counting as the
user's approval, instruction coverage). One finding is about the protocol, not
the SDK: an operation already performing at sign-out completed and reported
success, which Poppy leaves unspecified (raised on the Poppy tracker, #84).
0.36.x was not tested. The write-up, evidence and harness:
[studies/poppy-1.0](https://github.com/kkrlstrm/kirkaldy-conformance-kit/tree/main/studies/poppy-1.0).
- **The official company kit**: validate a company implementation, on these
SDKs or any other, with the protocol's
[kit/company](https://github.com/personal-agent-protocol/personal-agent-protocol/tree/main/kit/company).

See [Independent testing and the official kit](https://personal-agent-protocol.datalayer.tech/protocol/independent-testing/).

## Security boundary

PAP credentials belong to the host, not to an agent model or tool call.
Expand Down
50 changes: 50 additions & 0 deletions docs/docs/protocol/independent-testing.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
title: Independent testing and the official kit
sidebar_position: 5
---

# Independent testing and the official kit

Besides the SDKs' own [conformance report](./conformance-report.mdx), two outside
checks apply to these SDKs: an independent study of the Python SDK against
Poppy 1.0, and the protocol's official company kit, which anyone can run against
their own implementation.

## An independent study of the Python SDK 0.35.0

In November 2026 an independent tester of agent infrastructure ran the
**Python SDK 0.35.0, self-hosted and unmodified**, through tests of Poppy 1.0's
**Operations extension** and **sign-out**. The effects of each test were counted
by a separate recorder, not read from the SDK's own responses.

**Everything passed:**

| Area | What was tested | Result |
| --- | --- | --- |
| Retry after a lost response | The company answers again with the original response and performs the action once | Passed in 3 of 3 trials. A deliberately broken store performed the action twice, and the test caught it. |
| Sign-out | After sign-out, refresh is refused with `invalid_grant`, and an access token issued before sign-out is refused on the next request, with no effect | Passed |
| The agent side, six rules | Declines; revised terms after an agent's approval; lost responses; `Retry-After`; company text never counting as the user's approval; instruction coverage | Passed |

**One observation about the protocol, not the SDK:** an operation that was
already performing when the user signed out completed and reported success.
Poppy does not say what should happen in that case; it was raised on the Poppy
tracker (issue #84).

**What it does not cover:** the study tested 0.35.0. Version 0.36.x (the 1.0
draft as the default) has not been tested by it.

The write-up, its evidence and the harness are published:
[studies/poppy-1.0](https://github.com/kkrlstrm/kirkaldy-conformance-kit/tree/main/studies/poppy-1.0).
The same study proposes Operations checks for the official validator, which does
not test Operations yet, citing the lost-response test on this SDK as its
measured example.

## The official company kit

The Personal Agent Protocol project publishes a kit to validate a company's
implementation:
[kit/company](https://github.com/personal-agent-protocol/personal-agent-protocol/tree/main/kit/company)
in the protocol's repository. Run it against a company built on these SDKs, or on
any other implementation, before publishing its `poppy.json`. It is the
reference check of a company; this site's [conformance report](./conformance-report.mdx)
is the SDKs' own, run on every release.
Loading