feat: add cloudkittest, the contract kit

CHANGELOG.md

@@ -38,6 +38,27 @@ All notable changes to this project are documented here. The format follows
 - `UnsupportedError`, matching `ErrUnsupported`.
 - Policy tests: context first on every interface and registry method (C10), no package-level client
   or registry (C9), and no shipped exported name removed.
+- `RedactError`, which redacts an error's message and keeps its chain, so an adapter can return a
+  transport or provider error without leaking a credential and its caller still matches
+  `ErrEgressDenied` or `ErrDenied`.
+- Package `cloudkittest`, the contract kit. `Run` and `Check` run nine contract cases against any
+  `Provider` (transport only, nil transport, egress, read-only token, plan drift, destroy, redaction,
+  cloud-init and SSH), each over its own registry, a token generated for the run and no network;
+  `Report` gives each case's result.
+- `cloudkittest.Recorder`, an `http.RoundTripper` that replays one cassette, answers each interaction
+  once and in order, refuses any request the cassette does not hold (`ErrUnexpectedRequest`), and
+  reports interactions nothing used.
+- `cloudkittest.ParseRecordings` and `LoadRecordings`: every recorded and synthetic response is
+  checked against the provider's OpenAPI document, and a recording carrying an `Authorization`
+  header, a cookie, an API-key header or a value `Redact` would change is refused
+  (`ErrCredentialInRecording`).
+- `cloudkittest.ParseSpec`, `LoadSpec`, `Spec.ValidateResponse` and `Spec.Synthesize`: an OpenAPI 3
+  validator over a stated JSON Schema subset (`type`, `nullable`, `required`, `properties`,
+  `additionalProperties`, `items`, `enum`, local `$ref`) that refuses any other keyword
+  (`ErrUnsupportedKeyword`), and synthetic responses generated from the document's examples, marked
+  `Synthetic`.
+- Policy tests read `cloudkittest` too: no default HTTP client or transport, context first on its
+  hook signatures, and its shipped names preserved.
 
 ### Changed
 

README.md

@@ -66,6 +66,10 @@ Hosted at <https://opensource.simtabi.com/documentation/simtabi/cloudkit/>.
 - [Architecture](docs/architecture.md) — the rules every adapter follows, and why
 - [Release](docs/release.md) — how a version is tagged and released
 
+### Reference
+
+- [`cloudkittest`](docs/tools/cloudkittest.md) — the contract suite every adapter passes, the replaying recorder and the OpenAPI validator
+
 ## Contributing and security
 
 See [CONTRIBUTING.md](CONTRIBUTING.md). Report vulnerabilities privately, as

docs/architecture.md

@@ -36,11 +36,15 @@ its own engine adapters.
 | C10 | `context.Context` first on every call; the logger is injected |
 
 C1 to C7, C9 and C10 are implemented in this release; C8 arrives with the SSH path. A policy test
-in `src/internal/policy` scans every Go file for the symbols that would break C1, C4, C8, C9 and
-C10, with an exemption list that records the exact count it allows. Three more policy tests read
-the package's syntax: every interface method, `*Registry` method and exported function field takes
-a `context.Context` first (C10); no package-level client or registry exists (C9); and no exported
-name the package has shipped disappears.
+in `src/internal/policy` scans every Go file of the module, `cloudkittest` included, for the
+symbols that would break C1, C4, C8, C9 and C10, with an exemption list that records the exact count
+it allows. Three more policy tests read the syntax of `cloudkit` and `cloudkittest`: every interface
+method, `*Registry` method, exported function type and exported function field takes a
+`context.Context` first (C10); no package-level client or registry exists (C9); and no exported
+name either package has shipped disappears.
+
+Every adapter must also pass the contract suite in `cloudkittest`: nine cases, one per rule a
+provider could break, run over recorded responses. See [`cloudkittest`](tools/cloudkittest.md).
 
 ## Why the caller owns the transport
 
@@ -75,6 +79,30 @@ token for the provider's object store. Retries never resend a non-idempotent req
 ambiguous failure, so a create is not sent twice; a 429 is retried because the provider did not act
 on it.
 
+## Why the contract suite replays spec-checked recordings
+
+An adapter is proven against what the provider actually sends, without spending money or needing a
+token on every pull request. Each case replays a cassette through a `Recorder`, which refuses any
+request the cassette does not hold, so an adapter cannot reach the network and cannot make a call
+nobody recorded. A case also fails when a recorded request goes unused, because the defect is often
+the call an adapter skipped, such as the second live-state read that catches drift.
+
+Recordings go stale, and hand-edited ones lie. So every response, recorded or synthetic, is checked
+against the provider's published OpenAPI document when the recordings load, and a fixture that no
+longer matches the schema does not load at all. A write path with no recording yet replays a
+response generated from the document's own examples, marked synthetic. The validator implements a
+small JSON Schema subset and refuses any keyword outside it, rather than skipping it: a schema
+check that quietly ignores part of the schema reports a pass it did not earn.
+
+A recording is committed to a public repository, so the loader refuses one that carries an
+`Authorization` header, a cookie, an API-key header, or any value the module's redactor would
+change. The recorder never compares or stores request headers, so the token cloudkit adds to each
+request has nowhere to land.
+
+Each case runs with a token generated for that run. The redaction case puts the token where real
+failures put it, in a proxy error that echoes the request and in a provider body that echoes the
+token, and then searches every error, log record, panic and URL for it.
+
 ## Why it is a library without a CLI framework
 
 Each consumer has its own command tree and configuration loader. A library that brought cobra or a
@@ -97,11 +125,12 @@ before it loads.
 | Path | Holds |
 |---|---|
 | `src/` | package `cloudkit`: transport, egress policy, secrets, errors, redaction, the registry, capabilities, plans and `Apply`, retries |
+| `src/cloudkittest/` | the contract suite (`Run`, `Check`), the replaying `Recorder`, recordings and the OpenAPI validator |
 | `src/internal/policy/` | repository-wide tests: forbidden symbols, workflow guards |
 
 Planned packages are added when they have code: `providers/` (wave 1: Hetzner, DigitalOcean,
-Linode, Vultr; plus `generic` for any SSH host), `ssh/`, `cloudinit/`, `transport/`, `descriptor/`
-and `cloudkittest/`.
+Linode, Vultr; plus `generic` for any SSH host), `ssh/`, `cloudinit/`, `transport/` and
+`descriptor/`.
 
 ---
 

docs/tools/cloudkittest.md

@@ -0,0 +1,152 @@
+# `cloudkittest`
+
+`cloudkittest` (`github.com/simtabi/cloudkit/src/cloudkittest`) is the contract kit: it runs the nine
+contract cases every adapter must pass against any `cloudkit.Provider`, over recordings that have been
+checked against the provider's OpenAPI document, with nothing reaching the network.
+
+## Running the suite
+
+`cloudkittest.Run(t, suite)` runs each case as a subtest. `cloudkittest.Check(ctx, suite)` runs them
+and returns a `Report` without a `*testing.T`, for code that must refuse an adapter whose recordings
+fail, such as the descriptor loader.
+
+```go
+func TestHetznerContract(t *testing.T) {
+	fsys := os.DirFS("testdata")
+	spec, err := cloudkittest.LoadSpec(fsys, "hetzner.openapi.json")
+	if err != nil {
+		t.Fatal(err)
+	}
+	recs, err := cloudkittest.LoadRecordings(fsys, "hetzner.recordings.json", spec)
+	if err != nil {
+		t.Fatal(err)
+	}
+	cloudkittest.Run(t, cloudkittest.Suite{
+		Factory:    hetzner.Factory(),
+		Recordings: recs,
+		Writes: []cloudkittest.Write{
+			{Name: "PlanCreate", Capability: cloudkit.CapServerCreate, Plan: planCreate},
+			{Name: "PlanDestroy", Capability: cloudkit.CapServerDestroy, Plan: planDestroy},
+		},
+		Drift: planCreate,
+		Destroy: &cloudkittest.Destroy{Plan: planDestroy, Loses: []cloudkit.Resource{
+			{Kind: "server", ID: "42"}, {Kind: "volume", ID: "7"}, {Kind: "ipv4", ID: "9"},
+		}},
+		CloudInit: &cloudkittest.CloudInit{Render: render, Fields: fields},
+	})
+}
+```
+
+Each case builds its own `cloudkit.Registry` with a token generated for the run, no retries, dialing
+refused, and a logger that keeps every record. A case's failure messages never print that token, even
+when the adapter leaked it.
+
+## The nine cases
+
+| Case | Passes when |
+|---|---|
+| `transport-only` | Probe and Inventory reach the injected transport; with a transport that fails every request, every call fails; every write the probe allows has a `Write` in the suite |
+| `nil-transport` | `NewRegistry` refuses a nil round tripper and a nil dialer; `Factory.New` refuses an account with a nil client, or a client with no `Transport` (which would use the default one) |
+| `egress` | the adapter's client refuses a host outside `APIHosts` before the transport sees it; Probe and Inventory, redirected to an unlisted host, return an error matching `ErrEgressDenied`; the provider describes the same hosts as its factory |
+| `read-only-token` | with the read-only cassette, Probe reports every write as `Denied` and every `Write` returns `ErrDenied`, with no write request sent |
+| `plan-drift` | `Apply` of the `Drift` plan, after the live state moved, returns `ErrConflict` and sends no write |
+| `destroy` | the plan lists every resource in `Destroy.Loses`; `Apply` with nothing or the wrong name typed refuses without a request; with the names typed back it goes through |
+| `redaction` | behind a proxy that echoes the request and a provider that echoes the token, no error, log record or panic carries the token, and no URL does |
+| `cloud-init` | plain user data renders within `UserDataLimit`; a private key, bearer token or password in any field but `OneTimeToken` is refused; output above the limit is refused |
+| `ssh` | under every host-key policy but `accept-new`, a key never seen fails; no policy offers `password` or `keyboard-interactive` |
+
+A read-only adapter (no `Writes`) skips `plan-drift` and `destroy`. `cloud-init` is skipped only when
+`UserDataLimit` is 0, and fails when the provider takes user data but the suite has no `CloudInit`.
+`ssh` is skipped, with its reason, while `Suite.SSH` is nil. Every cassette a case replays must be
+used in full: a recorded request the adapter never sent fails the case, because skipping it (a
+live-state read, say) is often the defect.
+
+## Recordings
+
+A recordings file holds named cassettes, each an ordered list of request and response pairs:
+
+```json
+{
+  "provider": "fake",
+  "cassettes": {
+    "read-write": [
+      {"request": {"method": "GET", "url": "https://api.fake.test/v1/token"},
+       "response": {"status": 200, "headers": {"Content-Type": ["application/json"]}, "body": {"write": true}}}
+    ],
+    "destroy": [
+      {"request": {"method": "DELETE", "url": "https://api.fake.test/v1/resources/s1"},
+       "synthetic": {"status": 200, "example": "deleted"}}
+    ]
+  }
+}
+```
+
+| Cassette | Holds |
+|---|---|
+| `read-write` | Probe, then Inventory, with a token that may write |
+| `read-only` | Probe with a read-only token, then what each `Write` sends before it is refused |
+| `drift` | what the `Drift` plan sends, then the live-state read `Apply` makes, returning a different state |
+| `destroy` | what the `Destroy` plan sends, then the live-state read and the deletes of applying it |
+
+`ParseRecordings` is the only way to build `Recordings`, and it refuses:
+
+| Refused | Error |
+|---|---|
+| an `Authorization`, `Proxy-Authorization`, `Cookie`, `Set-Cookie`, `X-Auth-Token`, `X-Api-Key` or `Api-Key` header, in a request or a response | `ErrCredentialInRecording` |
+| a URL, header value or body that `cloudkit.Redact` would change (a bearer token, a credential query parameter, URL userinfo) | `ErrCredentialInRecording` |
+| a request or status the provider's document does not describe | `ErrNotInSpec` |
+| a body that breaks its response schema | `ErrSchemaViolation` |
+| unknown fields, an interaction with both or neither of `response` and `synthetic` | an error naming the cassette and position |
+
+A `synthetic` entry is generated by `Spec.Synthesize` from the document's examples and marked
+`Synthetic`, for a write path with no recording yet. The example is checked against its own
+schema, so a wrong example is refused like a wrong recording.
+
+## `Recorder`
+
+`NewRecorder(recs, cassette)` returns an `http.RoundTripper` over one cassette. A request is answered by
+the first unused interaction with the same method, URL (scheme, host, path, and query in any order) and
+JSON body; each interaction answers once, in order. Any other request returns an
+`*UnexpectedRequestError`, matching `ErrUnexpectedRequest`, and is sent nowhere. Headers are not
+compared, so the `Authorization` header cloudkit adds is ignored and never stored.
+
+| Method | Returns |
+|---|---|
+| `Requests()` | every request seen, URLs redacted, refused ones flagged `Unexpected` |
+| `Unexpected()` | the requests the cassette did not hold |
+| `Unused()` | the interactions nothing asked for |
+| `Err()` | every unexpected request as one error, or nil |
+
+Recording new responses from a live provider is the job of a scripted recording run, which arrives with
+the first provider adapter.
+
+## The OpenAPI validator
+
+`ParseSpec` reads an OpenAPI 3 document in JSON and compiles every schema a response reaches.
+`Spec.ValidateResponse(method, url, status, header, body)` finds the server (scheme, host and base
+path), the path template (a literal segment beats a `{param}`), the method and the response (the exact
+status, then `2XX`, then `default`), and checks the body.
+
+| Keywords | Treatment |
+|---|---|
+| `type`, `nullable`, `required`, `properties`, `additionalProperties`, `items`, `enum`, `$ref` (within the document) | checked |
+| `description`, `title`, `example`, `default`, `format`, `deprecated`, `readOnly`, `writeOnly` | accepted, not checked |
+| anything else (`allOf`, `oneOf`, `pattern`, `minimum`, ...) | the document is refused with `ErrUnsupportedKeyword` |
+
+Refusing an unknown keyword, rather than skipping it, means a response is either checked in full or not
+loaded at all. When a provider's document needs a keyword, the validator grows to check it. Only JSON
+media types are read, and a response object may not itself be a `$ref`.
+
+## Errors
+
+| Error | Meaning |
+|---|---|
+| `ErrUnexpectedRequest` | a recorder was sent a request its cassette does not hold |
+| `ErrCredentialInRecording` | a recording carries a credential header or value |
+| `ErrSchemaViolation` | a body breaks its schema; `*SchemaError` names the JSON path |
+| `ErrNotInSpec` | a server, path, method, status or content type the document does not describe |
+| `ErrUnsupportedKeyword` | a schema keyword outside the subset |
+
+---
+
+[← Docs index](../../README.md#documentation)

src/client.go

@@ -74,11 +74,18 @@ type redactedError struct {
 func (e *redactedError) Error() string { return e.msg }
 func (e *redactedError) Unwrap() error { return e.err }
 
-// redactErr is how an error from outside the module (a provider, an adapter,
-// a credential source) leaves it.
-func redactErr(err error) error {
+// RedactError returns err with its message passed through [Redact], while
+// errors.Is and errors.As still see the original chain. An adapter returns
+// every error it did not build itself (a transport failure, a provider's
+// response) through it, so a credential a proxy or a provider echoed back
+// never leaves the module. It returns nil for a nil error.
+func RedactError(err error) error {
 	if err == nil {
 		return nil
 	}
 	return &redactedError{msg: Redact(err.Error()), err: err}
 }
+
+// redactErr is how an error from outside the module (a provider, an adapter,
+// a credential source) leaves it.
+func redactErr(err error) error { return RedactError(err) }

src/cloudkit_test.go

@@ -269,3 +269,25 @@ func TestEgressBoundsRedirectChains(t *testing.T) {
 		t.Fatalf("followed %d hops", len(rt.seen))
 	}
 }
+
+// RedactError strips credentials from the message and keeps the chain, so an
+// adapter can redact a transport error and its caller still matches
+// ErrEgressDenied.
+func TestRedactErrorKeepsTheChain(t *testing.T) {
+	if RedactError(nil) != nil {
+		t.Fatal("RedactError(nil) is not nil")
+	}
+	tok := rand.Text()
+	cause := &EgressError{Host: "elsewhere.test", Reason: "proxy said Authorization: Bearer " + tok}
+	err := RedactError(fmt.Errorf("probe: %w", cause))
+	if strings.Contains(err.Error(), tok) {
+		t.Errorf("the token survived: %q", err.Error())
+	}
+	if !errors.Is(err, ErrEgressDenied) {
+		t.Error("errors.Is no longer reaches ErrEgressDenied")
+	}
+	var eg *EgressError
+	if !errors.As(err, &eg) || eg.Host != "elsewhere.test" {
+		t.Errorf("errors.As lost the EgressError: %v", eg)
+	}
+}