diff --git a/CHANGELOG.md b/CHANGELOG.md index 41ddc58..be3bd5c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and - Add backward-compatible cost, timeout, and freshness metadata to agent-job results. - Add a default-deny action-control schema, deterministic evaluator, documentation, and synthetic dry-run and authorized-action examples. - Add a deterministic local Virtuals ACP public-evidence verification service with a bounded request schema, conforming job-result receipts, hidden offering metadata, timeout/retry behavior, and synthetic buyer/provider lifecycle fixtures. +- Add a network-free community-validation harness, bounded result schema, external tester protocol, feedback template, and unsent upstream-review and sponsorship-request drafts. ### Security diff --git a/README.md b/README.md index ab36f4e..83d303a 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,8 @@ For deterministic dry-run and default-deny approval decisions, see [docs/action- For the selected Phase 3 Virtuals ACP job, see [docs/virtuals-acp-evidence-service.md](docs/virtuals-acp-evidence-service.md), `schemas/acp-evidence-request-v1.schema.json`, and `agentic_commerce.acp_evidence`. Local mode simulates a buyer/provider evidence-verification lifecycle, emits a conforming job-result receipt, and performs no wallet, payment, signing, inference, or network operation. +For network-free external testing, public-safe result records, and draft upstream review questions, see [docs/community-validation.md](docs/community-validation.md), [docs/upstream-review-requests.md](docs/upstream-review-requests.md), and `scripts/run_community_validation.py`. Generated validation records are not uploaded automatically and default to no public-sharing approval. + For silent compatibility drift detection and bounded weekly project evidence, see [docs/automation.md](docs/automation.md). These helpers are read-only and never perform GitHub mutations. At the 2026-07-11 compatibility check, the datanet and pod catalogs were live. The documented public stats route returned HTTP 404, so `status` and `snapshot` correctly returned partial result code `2` while preserving catalog data. The upstream pods route also ignored its requested page size; the client applies the requested limit after a capped download. diff --git a/docs/README.md b/docs/README.md index ad13a64..86f84d1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,6 +5,8 @@ - [Agent job result schema](provenance-schemas.md#agent-job-result-v1) - [Dry-run and approval controls](action-controls.md) - [Virtuals ACP public-evidence reference service](virtuals-acp-evidence-service.md) +- [Community validation protocol](community-validation.md) +- [Upstream review request drafts](upstream-review-requests.md) - [Read-only maintenance automation](automation.md) - [Release process](releasing.md) diff --git a/docs/community-validation.md b/docs/community-validation.md new file mode 100644 index 0000000..f86b49d --- /dev/null +++ b/docs/community-validation.md @@ -0,0 +1,143 @@ +# Community validation protocol + +## Purpose + +Phase 4 tests whether the toolkit is useful to people outside its maintainers. This protocol gives external users a reproducible, network-free way to exercise the Phase 3 Virtuals ACP reference service and return bounded public evidence. + +Running the protocol does not contact Reppo, Virtuals, GitHub, a wallet, an inference provider, or any other remote service. Nothing is uploaded automatically. + +## What the harness checks + +`scripts/run_community_validation.py` exercises four synthetic behaviors: + +1. the published evidence bundle produces a passing receipt and exit code `0`; +2. a provenance mismatch produces a completed finding and exit code `2`; +3. bounded retry exhaustion produces a sanitized failed receipt and exit code `1`; +4. the passing flow remains local, zero-cost, and explicitly simulated. + +The result conforms to `schemas/community-validation-result-v1.schema.json`. It intentionally excludes usernames, hostnames, architecture, local paths, wallets, accounts, raw logs, and environment variables. + +## Tester protocol + +### 1. Use a clean public checkout + +Clone the public repository or update an existing public-only checkout. Do not run this protocol from a directory containing copied private artifacts. + +Record the public revision: + +```bash +git rev-parse --short=12 HEAD +``` + +### 2. Run the complete repository checks + +```bash +PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python3.11 -B \ + -m unittest discover -s tests -q +python3 scripts/check_public_boundary.py . +git diff --check +python3 -m compileall -q src scripts tests +``` + +No package installation is required for the runtime harness. The complete schema-contract tests require the repository's documented development environment. + +### 3. Generate the bounded validation result + +The maintainer should assign a non-personal validation ID such as `community:test-run:001`. Do not put a name, email address, account ID, wallet address, hostname, or local directory in the ID. + +```bash +PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python3.11 -B \ + scripts/run_community_validation.py \ + --validation-id community:test-run:001 \ + --revision YOUR_PUBLIC_REVISION \ + --pretty > community-validation-result.json +``` + +Expected command exit code: `0`. + +Expected result: + +- top-level `status` is `passed`; +- all four checks have `status: pass`; +- `errors` is empty; +- both `sharing` flags are `false`. + +The false sharing flags are deliberate. Generation is not publication approval. + +### 4. Review before sharing + +Open `community-validation-result.json` and verify that it contains no: + +- local or home-directory path; +- username, hostname, email address, or contact detail; +- account, wallet, transaction, or payment identifier; +- credential, token, cookie, environment variable, or raw log; +- private project name, communication, research, or report. + +After completing that human review, a tester who chooses to submit the record may set: + +```json +"sharing": { + "reviewedForSensitiveData": true, + "publicSubmissionApproved": true +} +``` + +Do not change those flags on another person's behalf. + +### 5. Provide bounded feedback + +Use `examples/community-validation/feedback-template.md`. A tester may submit the reviewed JSON and feedback through a public GitHub issue or pull request when the maintainer opens an approved collection channel. + +Until such a channel exists, keep the result local. Do not send it through private communications for later copying into this repository. + +## What counts as an external test user + +A Phase 4 test counts only when all of the following are true: + +- the person is not acting as the toolkit maintainer for that run; +- they run the protocol against a public revision; +- the machine-readable result passes the schema contract; +- they provide at least one concrete observation in the feedback template; +- they intentionally approve any public submission; +- the maintainer records a public evidence link after review. + +Expressions of interest, maintainers running the harness three times, copied results, or unpublished private feedback do not count toward the roadmap target. + +Recommended tester mix: + +1. one schema consumer; +2. one command-line user working from a fresh checkout; +3. one ACP builder able to assess the offering and lifecycle mapping. + +## Maintainer evidence table + +Do not add a row until public evidence exists. + +| Validation ID | Public revision | Tester perspective | Result evidence | Feedback evidence | Counted | +|---|---|---|---|---|---| + +The table must link only to intentionally public artifacts. It must not record private contact details or summarize private conversations. + +## Decision criteria + +After at least three qualifying tests and upstream review attempts, record one outcome: + +- **continue** — repeat use and a concrete next integration need are demonstrated; +- **narrow** — schemas or inspection are useful, but a live ACP provider is not justified; +- **archive** — external reproduction or demand is not demonstrated. + +Inference credits alone cannot determine the outcome. + +## Sponsorship gate + +Do not request inference sponsorship until: + +- three external test users have qualifying public evidence; +- Reppo and Virtuals review requests have been made through approved public channels; +- a real inference-backed next step is identified; +- expected usage has been measured or bounded without inventing demand. + +The current local service uses no inference, so it provides no evidence for a credit amount. Any later request should state an exact 60-day term, a hard total cap, periodic caps, permitted public workloads, data-retention terms, revocation conditions, expiry behavior, and the absence of any assumed token launch, exclusivity, wallet custody, or private-data access. + +Record the outcome even if no applicable program exists, terms are unsuitable, or sponsorship is unnecessary. diff --git a/docs/upstream-review-requests.md b/docs/upstream-review-requests.md new file mode 100644 index 0000000..16bdace --- /dev/null +++ b/docs/upstream-review-requests.md @@ -0,0 +1,155 @@ +# Upstream review request drafts + +These are review drafts, not evidence that outreach occurred. Sending a request, opening an issue, or submitting a pull request is a separate human-approved action. + +Before using either draft: + +1. verify all linked artifacts are present on the public default branch; +2. rerun the relevant live compatibility check; +3. remove observations that are stale or cannot be reproduced; +4. use an upstream project's preferred public contribution channel; +5. record only public response links in this repository. + +## Reppo builder review + +### Suggested title + +```text +Feedback request: public inspector behavior and portable provenance contracts +``` + +### Draft + +```text +We maintain the independent Agentic Commerce Toolkit, which includes a +standard-library, read-only inspector for Reppo's documented public catalog +endpoints and portable source-manifest / agent-job-result schemas. + +Project: +https://github.com/nccrypto/agentic-commerce-toolkit + +Inspector documentation: +https://github.com/nccrypto/agentic-commerce-toolkit/blob/main/docs/reppo-inspector.md + +Provenance contracts: +https://github.com/nccrypto/agentic-commerce-toolkit/blob/main/docs/provenance-schemas.md + +We would value upstream review of four bounded questions: + +1. Are the public endpoint boundaries and partial-result semantics still + consistent with the intended public API? +2. Is preserving upstream objects inside a stable outer envelope preferable + to normalizing their internal field names? +3. Are the source-manifest and agent-job-result contracts useful for public + agent workflows consuming Reppo data? +4. Did our freshly reproduced compatibility observations identify a current + documentation or API-behavior gap that should be reported separately? + +The inspector has no authentication, write, wallet, or transaction support. +Its CI tests use synthetic fixtures and do not contact live services. + +We are requesting technical feedback, not endorsement. We will record any +limitations or corrections in the public project. +``` + +### Required fresh evidence + +Do not mention an unavailable route, ignored pagination parameter, or other behavior unless it has been rerun immediately before outreach and recorded without private logs or credentials. + +## Virtuals ACP builder review + +### Suggested title + +```text +Feedback request: local ACP public-evidence provider and receipt mapping +``` + +### Draft + +```text +We maintain the independent Agentic Commerce Toolkit and have implemented a +network-free Virtuals ACP reference provider for one bounded job: public +evidence verification. + +Project: +https://github.com/nccrypto/agentic-commerce-toolkit + +Reference-service documentation: +https://github.com/nccrypto/agentic-commerce-toolkit/blob/main/docs/virtuals-acp-evidence-service.md + +Synthetic offering: +https://github.com/nccrypto/agentic-commerce-toolkit/blob/main/examples/virtuals-acp-evidence/offering-v1.example.json + +The buyer supplies a source manifest plus an agent-job result. The provider +checks contract shape, provenance linkage, and public HTTPS source policy, +then returns the existing agent-job-result contract as a machine-readable +receipt. + +Local mode simulates open → budget_set → funded → submitted → completed, but +performs no ACP authentication, registration, signing, wallet operation, +payment, settlement, inference, or network request. The offering example is +hidden and has not been registered. + +We would value review of five bounded questions: + +1. Do the requirement and deliverable shapes map cleanly to the current ACP + offering model? +2. Is reusing a structured agent-job result as the provider receipt practical? +3. Does the documentation distinguish simulated lifecycle states clearly + enough from live on-chain settlement? +4. What boundary would you recommend for a future adapter to acp-cli or + acp-node-v2 while keeping the verifier itself deterministic and testable? +5. Is this provider-side evidence job sufficiently distinct from existing + evaluator/recompute examples? + +We are requesting technical feedback, not endorsement, credits, listing, or +promotion. Corrections and limitations will be recorded publicly. +``` + +## Upstream contribution rule + +A review request is not permission to create unrelated upstream work. Open an issue or pull request only when a fresh, reproducible observation identifies a concrete documentation, compatibility, or interoperability gap. If no useful contribution is warranted, record that outcome instead of manufacturing activity. + +## Review evidence record + +Do not populate this table with private messages or contact information. + +| Upstream | Public request URL | Public response URL | Material correction or recommendation | Status | +|---|---|---|---|---| +| Reppo | — | — | — | not sent | +| Virtuals | — | — | — | not sent | + +## Bounded 60-day sponsorship request outline + +This outline is intentionally incomplete until community testing demonstrates an inference-backed need. + +```text +Subject: Bounded 60-day public integration validation request + +Shipped evidence: +- public repository and release/commit; +- qualifying community-validation evidence; +- upstream technical review links; +- exact proposed inference-backed workload; +- measured or conservatively bounded usage. + +Requested term: +- exactly 60 days; +- exact total credit cap: TO BE DERIVED FROM MEASURED USAGE; +- exact daily or weekly cap: TO BE DERIVED FROM MEASURED USAGE; +- permitted use: public toolkit validation workload only; +- automatic expiry after the term; +- no production-wallet custody or private-data processing. + +Terms to clarify: +- eligible models and endpoints; +- telemetry and data retention; +- rate limits and revocation; +- unused-credit expiry; +- attribution requirements; +- whether exclusivity, token launch, deposits, or other obligations apply. + +The toolkit does not assume approval, renewal, endorsement, or future credits. +``` + +Record one public outcome: accepted, declined, no applicable program, unsuitable terms, unnecessary, or deferred for insufficient demand. diff --git a/examples/README.md b/examples/README.md index f01c248..958bae8 100644 --- a/examples/README.md +++ b/examples/README.md @@ -5,5 +5,6 @@ - [Agent job result](agent-job-result/README.md) - [Dry-run and approval controls](action-control/README.md) - [Virtuals ACP public-evidence service](virtuals-acp-evidence/README.md) +- [Community validation](community-validation/README.md) Tested, public-source-only examples live here. diff --git a/examples/community-validation/README.md b/examples/community-validation/README.md new file mode 100644 index 0000000..9e2eb40 --- /dev/null +++ b/examples/community-validation/README.md @@ -0,0 +1,20 @@ +# Community validation examples + +Phase 4 uses these public-safe artifacts to collect reproducible external testing evidence: + +- `test-result-v1.example.json` — deterministic output from the network-free validation harness; +- `feedback-template.md` — bounded qualitative feedback with an explicit public-sharing review. + +Generate a fresh local result from the public repository root: + +```bash +PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python3.11 -B \ + scripts/run_community_validation.py \ + --validation-id community:test-run:001 \ + --revision YOUR_PUBLIC_REVISION \ + --pretty > community-validation-result.json +``` + +Generated records default both sharing flags to `false`. See [`docs/community-validation.md`](../../docs/community-validation.md) before changing or submitting them. + +The committed example is synthetic maintainer evidence that the harness works. It does not count as one of the three required external users and does not claim upstream review, endorsement, sponsorship, or live ACP execution. diff --git a/examples/community-validation/feedback-template.md b/examples/community-validation/feedback-template.md new file mode 100644 index 0000000..03ea2e9 --- /dev/null +++ b/examples/community-validation/feedback-template.md @@ -0,0 +1,72 @@ +# External tester feedback + +Use this template only after running the public [community validation protocol](../../docs/community-validation.md). + +Do not paste terminal transcripts, environment dumps, local paths, usernames, hostnames, email addresses, account or wallet identifiers, credentials, tokens, cookies, private communications, or private project details. + +## Validation evidence + +- Validation ID: +- Public toolkit revision: +- Reviewed machine-readable result attached or linked: +- Result `status`: +- All sharing flags intentionally set to `true`: yes / no + +## Tester perspective + +Select the closest public role. No employer, account, or personal identity is required. + +- [ ] Schema consumer +- [ ] Command-line user +- [ ] ACP builder +- [ ] Other public technical perspective: + +## Reproduction + +- Did the repository tests pass? yes / no / not run +- Did the public-boundary check pass? yes / no / not run +- Did the community harness exit with code `0`? yes / no +- Did all four harness checks pass? yes / no + +If not, describe the smallest reproducible public symptom without raw logs: + +> + +## Contract feedback + +1. Was the request contract understandable? + +> + +2. Was the receipt and its pass/fail distinction understandable? + +> + +3. Was simulated ACP lifecycle behavior clearly distinguished from live settlement? + +> + +4. What is the smallest improvement that would make the toolkit more useful? + +> + +## Demonstrated use + +Would you use any of these artifacts in a public project? + +- [ ] Source-manifest schema +- [ ] Agent-job result schema +- [ ] Action-control pattern +- [ ] Reppo inspector +- [ ] ACP public-evidence verifier +- [ ] None yet + +Optional bounded explanation: + +> + +## Public sharing approval + +- [ ] I reviewed this feedback for sensitive or private material. +- [ ] I intentionally approve publication of this feedback. +- [ ] I understand that public submissions may be retained in repository history. diff --git a/examples/community-validation/test-result-v1.example.json b/examples/community-validation/test-result-v1.example.json new file mode 100644 index 0000000..0f8a06c --- /dev/null +++ b/examples/community-validation/test-result-v1.example.json @@ -0,0 +1,42 @@ +{ + "checks": [ + { + "checkId": "passing-fixture", + "status": "pass", + "summary": "The published evidence fixture produced a successful pass receipt." + }, + { + "checkId": "finding-fixture", + "status": "pass", + "summary": "A synthetic provenance mismatch produced a completed fail verdict and exit code 2." + }, + { + "checkId": "failure-fixture", + "status": "pass", + "summary": "Bounded retry exhaustion produced a sanitized failed receipt and exit code 1." + }, + { + "checkId": "local-execution-boundary", + "status": "pass", + "summary": "The reference flow remained local, zero-cost, and explicitly simulated." + } + ], + "environment": { + "platform": "darwin", + "pythonVersion": "3.11.15" + }, + "errors": [], + "limitations": [ + "This harness exercises synthetic local fixtures and does not prove a live ACP job, payment, settlement, or upstream endorsement.", + "The generated record excludes raw logs and must still receive human sensitive-data review before public submission." + ], + "recordedAt": "2026-07-15T13:00:00Z", + "schemaVersion": "1.0", + "sharing": { + "publicSubmissionApproved": false, + "reviewedForSensitiveData": false + }, + "status": "passed", + "toolkitRevision": "4caed42", + "validationId": "example:community-validation:001" +} diff --git a/schemas/README.md b/schemas/README.md index 89b6dd3..4abc00c 100644 --- a/schemas/README.md +++ b/schemas/README.md @@ -10,5 +10,7 @@ - `../examples/action-control/` — synthetic dry-run and authorized-action examples validated in CI. - `acp-evidence-request-v1.schema.json` — bounded request envelope for the Virtuals ACP public-evidence verification job. - `../examples/virtuals-acp-evidence/` — synthetic request, receipt, and hidden offering examples validated in CI. +- `community-validation-result-v1.schema.json` — bounded, public-safe result contract for external validation runs. +- `../examples/community-validation/test-result-v1.example.json` — synthetic community-validation result validated in CI. Versioned JSON schemas for additional safety patterns will live here. diff --git a/schemas/community-validation-result-v1.schema.json b/schemas/community-validation-result-v1.schema.json new file mode 100644 index 0000000..36bf077 --- /dev/null +++ b/schemas/community-validation-result-v1.schema.json @@ -0,0 +1,137 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://raw.githubusercontent.com/nccrypto/agentic-commerce-toolkit/main/schemas/community-validation-result-v1.schema.json", + "title": "Agentic Commerce Community Validation Result v1", + "description": "A bounded, public-safe record of an external user's network-free toolkit validation run.", + "type": "object", + "additionalProperties": false, + "required": [ + "schemaVersion", + "validationId", + "recordedAt", + "toolkitRevision", + "environment", + "status", + "checks", + "errors", + "limitations", + "sharing" + ], + "properties": { + "schemaVersion": {"const": "1.0"}, + "validationId": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9._:-]{2,127}$", + "description": "Stable public identifier that does not contain a username, account, wallet, host, or local path." + }, + "recordedAt": {"type": "string", "format": "date-time"}, + "toolkitRevision": { + "type": "string", + "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$", + "description": "Public tag, branch label, or commit identifier tested." + }, + "environment": { + "type": "object", + "additionalProperties": false, + "required": ["pythonVersion", "platform"], + "properties": { + "pythonVersion": { + "type": "string", + "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$" + }, + "platform": { + "enum": ["linux", "darwin", "win32", "other"] + } + } + }, + "status": {"enum": ["passed", "failed"]}, + "checks": { + "type": "array", + "minItems": 1, + "maxItems": 10, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["checkId", "status", "summary"], + "properties": { + "checkId": { + "type": "string", + "pattern": "^[a-z][a-z0-9._-]{1,63}$" + }, + "status": {"enum": ["pass", "fail"]}, + "summary": { + "type": "string", + "minLength": 1, + "maxLength": 500 + } + } + } + }, + "errors": { + "type": "array", + "maxItems": 10, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["code", "message"], + "properties": { + "code": { + "type": "string", + "pattern": "^[A-Z][A-Z0-9_]{1,63}$" + }, + "message": { + "type": "string", + "minLength": 1, + "maxLength": 500 + } + } + } + }, + "limitations": { + "type": "array", + "maxItems": 20, + "items": { + "type": "string", + "minLength": 1, + "maxLength": 500 + } + }, + "sharing": { + "type": "object", + "additionalProperties": false, + "required": ["reviewedForSensitiveData", "publicSubmissionApproved"], + "properties": { + "reviewedForSensitiveData": {"type": "boolean"}, + "publicSubmissionApproved": {"type": "boolean"} + }, + "description": "Generated results default to false. A human sets these flags only after reviewing the file before public submission." + } + }, + "allOf": [ + { + "if": { + "properties": {"status": {"const": "passed"}}, + "required": ["status"] + }, + "then": { + "properties": { + "checks": { + "items": { + "properties": {"status": {"const": "pass"}} + } + }, + "errors": {"maxItems": 0} + } + } + }, + { + "if": { + "properties": {"status": {"const": "failed"}}, + "required": ["status"] + }, + "then": { + "properties": {"errors": {"minItems": 1}} + } + } + ] +} diff --git a/scripts/run_community_validation.py b/scripts/run_community_validation.py new file mode 100644 index 0000000..f53cb00 --- /dev/null +++ b/scripts/run_community_validation.py @@ -0,0 +1,252 @@ +"""Run the network-free Phase 4 community validation protocol.""" + +from __future__ import annotations + +import argparse +import copy +import json +import re +import sys +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Callable, Sequence, TextIO + +ROOT = Path(__file__).parents[1] +SRC = ROOT / "src" +if str(SRC) not in sys.path: + sys.path.insert(0, str(SRC)) + +from agentic_commerce.acp_evidence import ( # noqa: E402 + TransientProviderError, + receipt_exit_code, + run_local_evidence_job, +) + +_VALIDATION_ID = re.compile(r"^[a-z0-9][a-z0-9._:-]{2,127}$") +_REVISION = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$") +_DATE_TIME = re.compile( + r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})$" +) +REQUEST_EXAMPLE = ( + ROOT / "examples" / "virtuals-acp-evidence" / "request-v1.example.json" +) + + +def utc_now() -> str: + return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") + + +def _platform() -> str: + return sys.platform if sys.platform in {"linux", "darwin", "win32"} else "other" + + +def _check(check_id: str, passed: bool, passed_summary: str, failed_summary: str) -> dict[str, str]: + return { + "checkId": check_id, + "status": "pass" if passed else "fail", + "summary": passed_summary if passed else failed_summary, + } + + +def _validate_inputs(validation_id: str, revision: str, recorded_at: str) -> None: + if not _VALIDATION_ID.fullmatch(validation_id): + raise ValueError("validation_id must be a bounded public identifier") + if not _REVISION.fullmatch(revision): + raise ValueError("revision must be a bounded public tag or commit identifier") + if not _DATE_TIME.fullmatch(recorded_at): + raise ValueError("recorded_at must be an RFC 3339 date-time") + datetime.fromisoformat(recorded_at.replace("Z", "+00:00")) + + +def build_validation_result( + validation_id: str, + revision: str, + recorded_at: str, + *, + request_path: Path = REQUEST_EXAMPLE, +) -> dict[str, Any]: + """Exercise pass, finding, and bounded provider-failure paths.""" + + _validate_inputs(validation_id, revision, recorded_at) + checks: list[dict[str, str]] = [] + errors: list[dict[str, str]] = [] + static_clock: Callable[[], str] = lambda: recorded_at + static_monotonic: Callable[[], float] = lambda: 100.0 + + try: + request = json.loads(request_path.read_text(encoding="utf-8")) + + passing = run_local_evidence_job( + copy.deepcopy(request), + clock=static_clock, + monotonic=static_monotonic, + ) + passing_data = passing.get("result", {}).get("data", {}) + pass_ok = ( + receipt_exit_code(passing) == 0 + and passing.get("status") == "succeeded" + and passing_data.get("verdict") == "pass" + and all(item.get("status") == "pass" for item in passing_data.get("checks", [])) + ) + checks.append( + _check( + "passing-fixture", + pass_ok, + "The published evidence fixture produced a successful pass receipt.", + "The published evidence fixture did not produce the expected pass receipt.", + ) + ) + + mismatched = copy.deepcopy(request) + mismatched["jobResult"]["provenance"]["manifestId"] = "example:community-mismatch" + finding = run_local_evidence_job( + mismatched, + clock=static_clock, + monotonic=static_monotonic, + ) + finding_data = finding.get("result", {}).get("data", {}) + finding_ok = ( + receipt_exit_code(finding) == 2 + and finding.get("status") == "succeeded" + and finding_data.get("verdict") == "fail" + and any( + item.get("checkId") == "provenance-linkage" + and item.get("status") == "fail" + for item in finding_data.get("checks", []) + ) + ) + checks.append( + _check( + "finding-fixture", + finding_ok, + "A synthetic provenance mismatch produced a completed fail verdict and exit code 2.", + "The synthetic provenance mismatch did not produce the expected bounded finding.", + ) + ) + + def transient_failure(_candidate: Any) -> Any: + raise TransientProviderError("synthetic detail must not be returned") + + failed = run_local_evidence_job( + copy.deepcopy(request), + verifier=transient_failure, + max_attempts=2, + clock=static_clock, + monotonic=static_monotonic, + ) + failure_rendered = json.dumps(failed, sort_keys=True) + failure_ok = ( + receipt_exit_code(failed) == 1 + and failed.get("status") == "failed" + and failed.get("result") is None + and failed.get("errors", [{}])[0].get("code") == "RETRY_EXHAUSTED" + and "synthetic detail" not in failure_rendered + ) + checks.append( + _check( + "failure-fixture", + failure_ok, + "Bounded retry exhaustion produced a sanitized failed receipt and exit code 1.", + "The retry-exhaustion path did not produce the expected sanitized failure receipt.", + ) + ) + + local_only_ok = ( + passing.get("cost") + == {"amount": "0", "currency": "USD", "basis": "measured"} + and passing_data.get("mode") == "local" + and [item.get("status") for item in passing_data.get("lifecycle", [])] + == ["open", "budget_set", "funded", "submitted", "completed"] + ) + checks.append( + _check( + "local-execution-boundary", + local_only_ok, + "The reference flow remained local, zero-cost, and explicitly simulated.", + "The reference flow did not preserve its expected local execution boundary.", + ) + ) + except Exception: + checks = [ + _check( + "harness-execution", + False, + "The community validation harness completed.", + "The harness could not complete using the published fixture.", + ) + ] + errors.append( + { + "code": "HARNESS_ERROR", + "message": "The harness failed without exposing local exception or path details.", + } + ) + + failed_checks = [item for item in checks if item["status"] == "fail"] + if failed_checks and not errors: + errors.append( + { + "code": "VALIDATION_CHECK_FAILED", + "message": f"{len(failed_checks)} bounded community validation check(s) failed.", + } + ) + + return { + "schemaVersion": "1.0", + "validationId": validation_id, + "recordedAt": recorded_at, + "toolkitRevision": revision, + "environment": { + "pythonVersion": ".".join(str(part) for part in sys.version_info[:3]), + "platform": _platform(), + }, + "status": "failed" if failed_checks else "passed", + "checks": checks, + "errors": errors, + "limitations": [ + "This harness exercises synthetic local fixtures and does not prove a live ACP job, payment, settlement, or upstream endorsement.", + "The generated record excludes raw logs and must still receive human sensitive-data review before public submission.", + ], + "sharing": { + "reviewedForSensitiveData": False, + "publicSubmissionApproved": False, + }, + } + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--validation-id", required=True) + parser.add_argument("--revision", required=True) + parser.add_argument("--recorded-at", default=None) + parser.add_argument("--pretty", action="store_true") + return parser + + +def main( + argv: Sequence[str] | None = None, + *, + stdout: TextIO | None = None, + clock: Callable[[], str] = utc_now, +) -> int: + args = build_parser().parse_args(argv) + output = stdout or sys.stdout + try: + result = build_validation_result( + args.validation_id, + args.revision, + args.recorded_at or clock(), + ) + except ValueError as exc: + build_parser().error(str(exc)) + options: dict[str, Any] = {"sort_keys": True} + if args.pretty: + options["indent"] = 2 + else: + options["separators"] = (",", ":") + print(json.dumps(result, **options), file=output) + return 0 if result["status"] == "passed" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_community_validation.py b/tests/test_community_validation.py new file mode 100644 index 0000000..55fca84 --- /dev/null +++ b/tests/test_community_validation.py @@ -0,0 +1,101 @@ +import io +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).parents[1] +SPEC = importlib.util.spec_from_file_location( + "run_community_validation", ROOT / "scripts" / "run_community_validation.py" +) +if SPEC is None or SPEC.loader is None: + raise RuntimeError("Unable to load the community validation harness") +HARNESS = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(HARNESS) +build_validation_result = HARNESS.build_validation_result +main = HARNESS.main + + +class CommunityValidationTests(unittest.TestCase): + def test_harness_exercises_pass_finding_failure_and_local_boundary(self): + result = build_validation_result( + "example:community-validation:001", + "4caed42", + "2026-07-15T13:00:00Z", + ) + + self.assertEqual(result["status"], "passed") + self.assertEqual( + [check["checkId"] for check in result["checks"]], + [ + "passing-fixture", + "finding-fixture", + "failure-fixture", + "local-execution-boundary", + ], + ) + self.assertTrue(all(check["status"] == "pass" for check in result["checks"])) + self.assertEqual( + result["sharing"], + { + "reviewedForSensitiveData": False, + "publicSubmissionApproved": False, + }, + ) + self.assertNotIn(str(ROOT), json.dumps(result)) + + def test_main_prints_only_json_and_returns_zero(self): + stdout = io.StringIO() + + exit_code = main( + [ + "--validation-id", + "example:community-validation:002", + "--revision", + "4caed42", + "--pretty", + ], + stdout=stdout, + clock=lambda: "2026-07-15T13:05:00Z", + ) + + payload = json.loads(stdout.getvalue()) + self.assertEqual(exit_code, 0) + self.assertEqual(payload["recordedAt"], "2026-07-15T13:05:00Z") + self.assertTrue(stdout.getvalue().startswith("{\n")) + + def test_fixture_read_failure_is_sanitized(self): + with tempfile.TemporaryDirectory() as directory: + missing = Path(directory) / "missing-request.json" + result = build_validation_result( + "example:community-validation:003", + "4caed42", + "2026-07-15T13:10:00Z", + request_path=missing, + ) + + rendered = json.dumps(result) + self.assertEqual(result["status"], "failed") + self.assertEqual(result["errors"][0]["code"], "HARNESS_ERROR") + self.assertNotIn(str(missing), rendered) + self.assertNotIn(directory, rendered) + + def test_public_identifiers_are_bounded(self): + with self.assertRaises(ValueError): + build_validation_result( + "Contains Spaces", + "4caed42", + "2026-07-15T13:10:00Z", + ) + with self.assertRaises(ValueError): + build_validation_result( + "example:community-validation:004", + "revision with spaces", + "2026-07-15T13:10:00Z", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_schema_contract.py b/tests/test_schema_contract.py index 27b026c..5efb416 100644 --- a/tests/test_schema_contract.py +++ b/tests/test_schema_contract.py @@ -21,6 +21,8 @@ ACP_EVIDENCE_REQUEST_EXAMPLE = ROOT / "examples" / "virtuals-acp-evidence" / "request-v1.example.json" ACP_EVIDENCE_RECEIPT_EXAMPLE = ROOT / "examples" / "virtuals-acp-evidence" / "receipt-v1.example.json" ACP_EVIDENCE_OFFERING_EXAMPLE = ROOT / "examples" / "virtuals-acp-evidence" / "offering-v1.example.json" +COMMUNITY_VALIDATION_SCHEMA = ROOT / "schemas" / "community-validation-result-v1.schema.json" +COMMUNITY_VALIDATION_EXAMPLE = ROOT / "examples" / "community-validation" / "test-result-v1.example.json" def load_json(path): @@ -66,6 +68,34 @@ def test_acp_offering_example_is_bounded_and_nonexecuting(self): self.assertTrue(offering["isHidden"]) self.assertFalse(offering["requirements"].get("additionalProperties", True)) + def test_community_validation_example_conforms_and_defaults_to_no_sharing(self): + self.assert_conforms(COMMUNITY_VALIDATION_SCHEMA, COMMUNITY_VALIDATION_EXAMPLE) + result = load_json(COMMUNITY_VALIDATION_EXAMPLE) + + self.assertFalse(result["sharing"]["reviewedForSensitiveData"]) + self.assertFalse(result["sharing"]["publicSubmissionApproved"]) + + def test_community_validation_result_rejects_private_shaped_fields(self): + schema = load_json(COMMUNITY_VALIDATION_SCHEMA) + validator = Draft202012Validator(schema, format_checker=FormatChecker()) + result = load_json(COMMUNITY_VALIDATION_EXAMPLE) + result["localPath"] = "not allowed" + result["environment"]["hostname"] = "not allowed" + + self.assertFalse(validator.is_valid(result)) + + def test_community_validation_status_matches_checks_and_errors(self): + schema = load_json(COMMUNITY_VALIDATION_SCHEMA) + validator = Draft202012Validator(schema, format_checker=FormatChecker()) + + false_pass = load_json(COMMUNITY_VALIDATION_EXAMPLE) + false_pass["checks"][0]["status"] = "fail" + self.assertFalse(validator.is_valid(false_pass)) + + unexplained_failure = load_json(COMMUNITY_VALIDATION_EXAMPLE) + unexplained_failure["status"] = "failed" + self.assertFalse(validator.is_valid(unexplained_failure)) + def test_action_control_schema_enforces_default_deny(self): schema = load_json(ACTION_CONTROL_SCHEMA) validator = Draft202012Validator(schema, format_checker=FormatChecker())