Skip to content

Add conformance CI workflow (fault + prod lanes) [DEV-1160] - #17

Merged
pavel-y-ivanov merged 2 commits into
mainfrom
pavel/dev-1160-conformance-ci
Jul 13, 2026
Merged

Add conformance CI workflow (fault + prod lanes) [DEV-1160]#17
pavel-y-ivanov merged 2 commits into
mainfrom
pavel/dev-1160-conformance-ci

Conversation

@pavel-y-ivanov

@pavel-y-ivanov pavel-y-ivanov commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds .github/workflows/conformance.yml (DEV-1160): per-PR conformance CI running the raindrop-sdk-harness against this repo's Rust conformance driver (conformance/, DEV-1145). Mirrors the hardened workflow already landed in python-sdk, raindrop-js, and java-sdk.

Two lanes

  • fault-lane — every PR. Builds the standalone driver crate (cargo build --manifest-path conformance/Cargo.toml, binary at conformance/target/debug/raindrop-conformance-driver), starts the harness's local capture/fault server, and runs the full scenario corpus against it. No production calls; the only secret involved is the HARNESS_READ_TOKEN org secret used to check out the private harness repo, with a loud fork-safe skip when it's unavailable.
  • prod-lane — non-fork PRs with credentials configured. Drives the SDK against the real Raindrop ingest API and asserts records via the public Query API using the RAINDROP_WRITE_KEY / RAINDROP_QUERY_API_KEY org secrets (dedicated test org). Endpoints and credentials are scoped to the single step that needs them; skips loudly when secrets are missing.

Hardening (same pattern as the sibling SDKs)

  • Harness pinned to invisible-tools/raindrop-sdk-harness@b162d84a4c25b7203173f99e595de7ba88a57960 (main @ 2026-07-13, 27 scenarios); the bump procedure is documented in the workflow header.
  • All actions pinned to full commit SHAs (matching this repo's ci.yml convention, incl. the same dtolnay/rust-toolchain stable pin); persist-credentials: false on every checkout; permissions: contents: read; concurrency group with cancel-in-progress; 15-minute job timeouts.
  • continue-on-error: true on the runner steps while the suite beds in (non-blocking); each lane uploads a report JSON artifact and writes a full job summary. Ratchet baseline is the existing conformance/failures.txt.
  • Swatinem/rust-cache scoped to the conformance crate (workspaces: conformance) caches the cargo registry + the crate's target dir, keyed on its lockfile, to keep runtime sane.

Notes

  • The new workflow will not run on this PR itself (it only exists on the head ref); existing CI covers this diff. First real run happens on the next PR after merge.
  • Requires human approval — repo ruleset blocks self-approval; conformance CI-only diff (single new workflow file, no source changes).

🤖 Generated with Claude Code


Note

Medium Risk
Prod lane uses org secrets against live Raindrop APIs (scoped to one step), but no SDK/runtime code changes; main risk is CI misconfiguration or secret exposure patterns, mitigated by gating and persist-credentials: false.

Overview
Adds .github/workflows/conformance.yml, a PR-only workflow that runs the pinned raindrop-sdk-harness against the Rust conformance/ driver (cargo build --manifest-path conformance/Cargo.toml).

fault-lane builds the harness local server + runner, starts the fault capture server on 127.0.0.1:8787, and runs scenarios with --lane fault and conformance/failures.txt ratchet. prod-lane runs --lane prod against real ingest/query APIs when HARNESS_READ_TOKEN, RAINDROP_WRITE_KEY, and RAINDROP_QUERY_API_KEY are set; prod credentials are limited to the runner step.

Both jobs use gate steps for fork-safe loud skips, SHA-pinned actions, persist-credentials: false, concurrency cancel, rust-cache on conformance, job summaries, and report artifacts. Runner steps use continue-on-error: true so conformance is visible but not blocking yet.

Reviewed by Cursor Bugbot for commit 7886b2e. Bugbot is set up for automated code reviews on this repo. Configure here.

Runs the raindrop-sdk-harness against the Rust conformance driver
(conformance/, DEV-1145) on every pull request, mirroring the hardened
pattern from python-sdk/raindrop-js/java-sdk:

- fault-lane: local capture/fault server only, no credentials beyond the
  HARNESS_READ_TOKEN org secret; loud fork-safe skip.
- prod-lane: real Raindrop ingest + Query API via org secrets
  (RAINDROP_WRITE_KEY / RAINDROP_QUERY_API_KEY), step-scoped env,
  loud skip when secrets are unavailable.
- Harness pinned to invisible-tools/raindrop-sdk-harness@b162d84a
  (main @ 2026-07-13); bump procedure documented in the header.
- Actions SHA-pinned, persist-credentials: false, concurrency
  cancel-in-progress, 15min timeouts, continue-on-error while the suite
  beds in, report JSON artifact + job summary per lane.
- Swatinem/rust-cache scoped to the standalone conformance crate keeps
  registry + target caching sane.

DEV-1160

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 522fb32. Configure here.

pull_request:

permissions:
contents: read

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Artifact upload lacks token permission

Medium Severity

The workflow caps GITHUB_TOKEN at contents: read, but both lanes call actions/upload-artifact under if: always(). Uploading workflow artifacts needs write access to Actions (typically actions: write); with only contents: read, those upload steps often fail with “Resource not accessible by integration,” so report artifacts may never be published even when the runner succeeds.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 522fb32. Configure here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refuted empirically: this PR's own run (29274896649) uploaded both artifacts successfully — conformance-prod-report and conformance-fault-report are live on the run — under exactly this permissions block. actions/upload-artifact v4 authenticates uploads with the runner's ACTIONS_RUNTIME_TOKEN, not the job's GITHUB_TOKEN, so contents: read is sufficient (and is the same block the merged python-sdk/java-sdk/raindrop-js conformance workflows have been uploading with all day). Keeping least-privilege as-is.

New scenarios since the previous pin (wrap-capture-attachments, the three
expect.traces span scenarios) are all status: experimental — they report
but never gate, so no ratchet changes are needed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pavel-y-ivanov
pavel-y-ivanov merged commit 2d44364 into main Jul 13, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants