Do we need an LLM-focused AP2 conformance test? Or is AP2 simple enough that any LLM can handle it?

[pad01g]

Hi AP2 maintainers—thanks for all the work so far.

I’m exploring conformance testing for LLM-driven AP2 agents. The A2A TCK validates transport/API compliance, but it doesn’t tell us whether an LLM actually follows AP2’s flow rules.

Why this matters (examples):

• Older GPT models don’t “know” AP2. To run AP2 accurately we must inject long prompts/RAG; as AP2 becomes richer, this gets brittle.

• We’d like a test that answers “Is this model/agent AP2-capable?” E.g., can it:

  • keep the Intent → Cart → Payment order,
  • avoid issuing a CartMandate until all price-impacting info is collected,
  • place Intent/Payment in Messages and Cart in an Artifact,
  • re-price when a coupon or invite code is added before Payment (i.e., understand the whole flow, not just local steps).

Questions

1. Is an LLM-oriented AP2 conformance test suite needed, or is that out of scope for AP2?
   Related doubt: Is AP2 intentionally simple enough that almost any LLM can be guided with minimal prompts—so a separate conformance suite wouldn’t add value?
2. If it is needed, where should such a spec live—in the AP2 repo, in a separate repo like “AP2-TCK”?

Thanks for your guidance.

[chopmob-cloud]

Useful question. Sharing an adjacent piece of infrastructure that may be relevant to how a conformance suite for LLM-driven AP2 agents could look in practice.

What exists today (adjacent to AP2)

I run a public x402 adversarial bench at agent-trust-bench.algovoi.co.uk (source: chopmob-cloud/algovoi-bench, MIT — filed for listing in solana-foundation/pay-skills (open PR, not yet merged)). It tests exactly the class of question you're asking: does the LLM-driven agent follow the protocol's flow rules under adversarial conditions? — but the protocol it tests today is x402 payment-gating, not AP2 specifically.

The shape is intentionally close to what an AP2 conformance suite would look like:

• 138 adversarial profiles across 30 categories — sequence-violation tests, premature-action tests, prompt-injection tests, unit-ambiguity tests, currency-drift tests, etc.

• Three persona fixtures (safe-agent / aggressive-agent / unbounded-agent) for reproducible cross-LLM comparison

• Public bench-runner that hammers any LLM against the full profile set, scores it on what fraction it correctly refuses / accepts / flags

• Structured event log so individual agent decisions are inspectable; pass/fail is interpretable, not just numeric

So an answer to your three example behaviours from the bench's existing test patterns:

| Your AP2 conformance question | What an analogous bench profile would test |

|---|---|

| "Keep the Intent → Cart → Payment order" | Send a 402 with a CartMandate BEFORE the prerequisite Intent. Correct response: agent refuses; failure: agent proceeds as if Intent was issued. |

| "Avoid issuing a CartMandate until all price-impacting info is collected" | Send a request where price-impacting info (e.g. coupon code field) is structurally present but empty. Correct: agent collects, then commits; failure: agent commits with null-priced cart. |

| "Place Intent/Payment in Messages and Cart in an Artifact" | Structural placement check — agent emits a Cart in a Message. Correct: agent emits in Artifact; failure: agent places Cart in Message. |

| "Re-price when a coupon code is added before Payment" | Multi-step state test — emit a Cart, then mutate the cart payload, then check the agent re-fetches before committing PaymentMandate. Correct: agent re-prices; failure: agent commits to stale Cart. |

These all map onto the same bench architecture: each one is a profile with an adversarial prompt + structural setup + a scoring function evaluating the agent's response.

Direct answers to your two questions

Q1: Is an LLM-oriented AP2 conformance test suite needed?

Yes, almost certainly. The "any LLM can handle AP2 with minimal prompting" framing is operationally untrue based on what we've seen against the analogous x402 surface. Even simple sequence-ordering questions ("must do A before B") have non-trivial failure rates on top-tier models without flow-specific prompting. As AP2 adds complexity (coupon repricing, intent expiry windows, mandate delegation chains, dispute flows), this gets brittle fast. A separate conformance suite is the only way to give implementers a deterministic answer to "is my agent integration AP2-correct, or am I shipping accidental flow violations?"

The A2A TCK validates transport, not agent flow conformance. Different layer; both needed.

Q2: Where should such a spec live?

A separate "AP2-TCK" repo with a few key properties:

1. Profile-based architecture — each conformance behaviour is one declarative profile (prompt + structural setup + expected refusal/acceptance/flag). Lets new behaviours be added by spec contributors, not just maintainers.

2. LLM-agnostic harness — bench-runner-style design that hits any agent endpoint (HTTP / A2A / MCP / direct OpenAI/Anthropic API), so the same conformance set runs against every implementation

3. Public canonical instance — a hosted endpoint that LLM teams can point at to self-test. The x402 bench is at agent-trust-bench.algovoi.co.uk; an AP2-TCK would deserve ap2-tck.googleapis.com or ap2-conformance.fido.org or equivalent

4. Versioned profile sets — pin each test against an AP2 spec version, so as the spec evolves the conformance behaviour evolves with it (not against an ever-moving target)

5. Public scoreboard, optional — opt-in transparency. Models / agent implementations that pass get a verifiable badge.

Worth filing separately from the AP2 repo since the TCK's release cadence will be faster than the spec's, and the TCK should be able to test against multiple AP2 versions simultaneously.

Concrete collaboration offer

If the AP2 team wants to spin up an AP2-TCK, I'm happy to:

1. Open-source the bench architecture (already MIT) and donate the harness + 138-profile fixture format as the starting point. The shape is general; the profiles are x402-specific today but the test-runner / persona / event-log / scoring infrastructure is protocol-agnostic.

2. Contribute initial AP2-flow profiles covering the four behaviours you mentioned, plus the common adversarial patterns (sequence-violation, premature-commit, structural-placement, multi-step state mutation, prompt-injection, unit-ambiguity, currency-drift).

3. Run the canonical hosted instance (similar to agent-trust-bench.algovoi.co.uk) so implementers have a public endpoint to self-test against. AlgoVoi already runs the bench infrastructure on the production VPC; adding an AP2-TCK alongside is incremental.

Donating the bench architecture forward gives the AP2 ecosystem a head start on this exact problem. It also matches the spirit of how the bench was originally built — as a public good for the x402 ecosystem to coordinate around agent-safety testing.

cc @hilarl @arian-gogani — the cross-protocol conformance question is something we touched on tangentially in the on-chain mandate binding thread and the four-class JCS canonicalisation work — a shared TCK is the operational complement to a shared envelope grammar.

Happy to do any or all of (1) (2) (3) if useful to the AP2 maintainers. Or to defer if the AP2 team has a different conformance-test direction in mind.

---

AlgoVoi (chopmob-cloud) -- Acquisition enquiries: https://docs.algovoi.co.uk/acquisition

[arian-gogani]

the cross-protocol conformance question is exactly right. the four-class JCS canonicalization work converging in x402 (x402-foundation/x402#2322) gives AP2 a ready-made foundation: one SHA-256(JCS(...)) rule across behavioral, regulatory, cryptographic, and observational evidence classes.

for AP2 specifically, the question is what happens after the agent completes a purchase. the payment proves the transaction occurred. what's missing is tamper-evident proof that the agent acted within its authorized scope -- that it bought what the user asked for, at the price it was authorized to pay, without modifying the cart after commitment.

nobulex produces bilateral Ed25519 receipts over JCS-canonical JSON for exactly this. the receipt chain accumulates into Trust Capital -- a credit score for the agent that gates future autonomy. an agent with high Trust Capital from consistently completing AP2 flows correctly gets higher spending limits and less friction. an agent that deviates loses access before damage spreads.

the TCK angle is interesting: a conformance suite that tests whether the agent's behavioral receipt matches the AP2 flow it was supposed to follow. not just "did it follow the protocol" but "can it prove it did."

happy to contribute AP2-shaped test profiles from the nobulex receipt side if the TCK moves forward.

repo: https://github.com/arian-gogani/nobulex

[chopmob-cloud]

@arian-gogani — that framing closes the loop. The bench tests whether the agent made the right decision under adversarial conditions; nobulex receipts test whether the agent can prove it did. Those are genuinely complementary primitives, not duplicates.

How the two layers compose for an AP2-TCK

The cleanest joint shape I can see:

1. Bench-runner emits an adversarial AP2 flow (e.g. CartMandate-before-Intent ordering violation, mid-flow coupon mutation, premature PaymentMandate commit). Agent under test receives the flow.
2. Agent makes a decision — accept, refuse, flag, ask-for-clarification.
3. Decision gets evaluated against the profile's expected outcome (the existing bench scoring layer).
4. Bilateral receipt gets co-signed — agent emits a nobulex-style Ed25519 receipt over the JCS-canonical decision payload. Receipt binds the agent's identity to the specific decision it made on the specific adversarial input.
5. Trust Capital updates based on the receipt chain — agents that consistently make correct decisions on the bench surface accumulate Trust Capital, which feeds back into AP2 mandate authorisation limits.

That's a TCK that does both "did the agent follow the protocol?" and "can the agent prove it did?" without the two layers needing to know about each other internally. The receipt is the audit trail; the bench is the test fixture; the Trust Capital is the consequence.

Where the four-class JCS work actually helps

The substrate convergence in x402#2322 / #2334 means a single canonical-bytes pipeline serves all four classes:

• Behavioral (your nobulex receipts) — Ed25519 over JCS(decision payload)
• Regulatory (tooloracle / feedoracle) — ES256K over JCS(attestation)
• Cryptographic (AlgoVoi audit-bundle) — HMAC over JCS(row), Falcon-1024 over JCS(mandate) for PQC paths
• Observational (Agent 402 Tape, when it lands) — signed receipts over JCS(observation window)

For an AP2-TCK that wants to compose all four — bench decision + behavioral receipt + cryptographic audit-bundle entry + regulatory compliance attestation, all bound to one canonical decision payload — the JCS rule means a single verifier walks all four signatures over the same byte sequence. No class-specific re-canonicalisation. That's the operational payoff of the cross-class convergence: the TCK can run one set of canonical-bytes through four signature primitives and report on all four independently.

Concrete shape if the AP2 team wants to move on this

Joint contribution would naturally be:

• AlgoVoi side — bench-runner architecture (MIT) + 138 profile fixtures (~30 categories) as the starting profile shape + canonical hosted instance for self-test
• Nobulex side — bilateral receipt scheme + Trust Capital accumulation rule + receipt verifier
• Joint — AP2-specific profile set (the four behaviours from the original question plus the adversarial patterns), receipt-binding to bench-runner output, public canonical scoreboard

Happy to coordinate via a dedicated thread (the AP2 repo or an ap2-tck repo if one gets stood up). Same async pattern we have running with PQSafe on the PQC cross-scheme fixture work — GitHub for technical lock-in, email for the higher-bandwidth scope alignment.

Deferring to the AP2 maintainers on whether this is the direction they want for a TCK, or whether they have a different conformance-test architecture in mind. The bench-and-receipt composition is a shape that emerges naturally from the existing infrastructure on both sides; happy to align it differently if AP2's TCK direction goes another way.

---

AlgoVoi (chopmob-cloud) -- Acquisition enquiries: https://docs.algovoi.co.uk/acquisition