diff --git a/CHANGELOG.md b/CHANGELOG.md index 19d4b38f..4ff93eb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Security +- Added tenant-scoped operator recovery for Stripe reconciliation dead letters: + owner/admin callers can inspect bounded backlog state and authorize one exact + manual retry with an idempotent evidence reference, preserving append-only + automatic attempt history, finite hashed leases, current-provider reconciliation, + cross-tenant non-disclosure, and immediate return to `dead_letter` after a failed + manual attempt without exposing provider secrets or arbitrary exception text. - Added a durable leased Stripe reconciliation worker that consumes verified-event triggers through server-owned Subscription-to-organization authority, hashes lease secrets at rest, prevents concurrent/stale completion, retries with bounded diff --git a/docs/doctoring/stripe-reconciliation-dead-letter-recovery.md b/docs/doctoring/stripe-reconciliation-dead-letter-recovery.md new file mode 100644 index 00000000..10fe5c13 --- /dev/null +++ b/docs/doctoring/stripe-reconciliation-dead-letter-recovery.md @@ -0,0 +1,89 @@ +# Stripe reconciliation dead-letter recovery + +## Status and scope + +This document describes **active pull-request behavior**, not protected-`develop` shipped truth. The recovery slice is stacked on the finite-lease Stripe reconciliation worker and remains non-integrable independently of that prerequisite stack. + +The bounded objective is operational: when verified Stripe reconciliation work has exhausted its automatic retry budget and reached durable `dead_letter`, an authorized workspace operator can inspect that tenant's backlog and explicitly retry one exact verified Event without resetting automatic retry history, deleting prior evidence, or treating webhook delivery order as current billing authority. + +## Decision + +ScopeWeave uses a separate manual recovery authority instead of resetting `billing_stripe_reconciliation_jobs.attempt_count` or reopening the automatic retry budget. + +For one dead-letter Event, an owner/admin supplies a bounded evidence reference such as an incident, ticket, or change identifier. ScopeWeave then: + +1. resolves the Event's Subscription and organization through existing normalized server-owned Stripe identity; +2. creates one new finite worker lease and monotonically increasing attempt number; +3. appends one immutable `billing_stripe_reconciliation_recoveries` authorization record linked to that exact attempt and authenticated actor; +4. re-fetches current Subscription/Invoice state through the existing authoritative reconciliation service; +5. atomically records either the resulting successful claim decision or a new terminal dead-letter outcome; and +6. returns a bounded non-secret receipt to the operator. + +A failed manual recovery does **not** enter another automatic retry cycle. It returns immediately to `dead_letter`. Another manual attempt requires a new explicit evidence reference. + +## Why this is safer than resetting retry state + +Resetting `attempt_count` would blur automatic and operator-authorized work, could collide with the append-only `(event_id, attempt_number)` attempt key, and would make the historical retry budget difficult to audit. Deleting prior attempts would be worse because it would destroy the evidence needed to explain why an operator recovery was necessary. + +The recovery table therefore stores only recovery-specific facts: recovery identity, Event/attempt identity, authenticated actor, bounded evidence reference, timestamps, terminal recovery outcome, sanitized error code, and optional claim-decision identity. Tenant identity remains normalized through Event trigger → Subscription → Customer → organization. Attempt timing/outcome remains on worker attempt history. Provider payloads, webhook raw bodies, lease plaintext, API secrets, session tokens, and membership authority are not copied into recovery evidence. + +## Idempotency and concurrency + +`(event_id, evidence_reference)` is unique. Replaying the same recovery request returns the already-durable receipt and does not perform another provider read or create another worker attempt. + +A new recovery is claimed only while the exact job is still `dead_letter` and its current attempt count matches the selected row. The job transition, new attempt, and recovery-authorization row share one named SQLite savepoint. If any mutation fails, ScopeWeave performs `ROLLBACK TO SAVEPOINT` and releases the savepoint only after rollback is confirmed, preserving the causal error and avoiding a partial operator authorization. + +Completion and failure also wrap the existing worker completion/failure operation inside the recovery savepoint. Because the worker uses a distinct nested savepoint, worker state, append-only attempt outcome, and recovery receipt commit or roll back together. + +## Lease and crash behavior + +The plaintext manual lease token is returned only inside the in-process recovery orchestration boundary. Durable worker state contains only its SHA-256 digest and finite expiration. + +If a process dies after manual claim and before resolution, the existing worker's expired-lease logic remains authoritative. Since a manual attempt number is already beyond the automatic attempt budget, lease expiry resolves back to terminal dead-letter evidence rather than silently reopening automatic retries. Replaying the same operator evidence can derive the dead-letter outcome from the linked append-only attempt. + +## Authorization and tenant isolation + +The HTTP surface is purpose-specific: + +- `GET /api/orgs/:id/billing/reconciliation/dead-letters` +- `POST /api/orgs/:id/billing/reconciliation/dead-letters/:eventId/retry` + +Authentication uses the existing JWT/PAT credential contract. Workspace owners/admins may inspect or retry only their own organization. The repository scopes dead-letter selection through normalized Subscription → Customer → organization identity; callers do not provide a Subscription ID. A caller who is authorized in one workspace but supplies an Event belonging to another receives a non-disclosing not-found result. + +Responses use `Cache-Control: no-store` and never expose lease tokens, Stripe secrets, raw provider bodies, webhook raw bytes, or arbitrary exception text. + +## Evidence reference contract + +The evidence reference is operator provenance and idempotency authority, not free-form incident content. It must be a non-empty control-free string no longer than 256 characters. Operators should use a stable identifier that an auditor can follow in the system of record, for example `INC-2026-0042` or a bounded change-ticket key, rather than copying sensitive incident narratives into ScopeWeave. + +## Failure handling + +Provider/persistence/policy exceptions may contain sensitive provider text. Only bounded machine-readable `stripe_*` error codes are retained; otherwise the outcome collapses to `stripe_reconciliation_recovery_failed`. + +If ScopeWeave cannot prove that worker completion/failure and recovery-receipt persistence changed atomically, the API fails closed as `stripe_reconciliation_recovery_state_uncertain`. It must not invent success or issue a speculative second provider call. + +## TDD and acceptance evidence + +The tests were introduced before the production recovery module/route existed. + +`tests/unit/stripe-reconciliation-dead-letter-recovery.test.mjs` exercises tenant-scoped listing, append-only attempt 5 → manual attempt 6, authoritative success, immediate dead-letter failure, same-evidence replay, new-evidence retry, input bounds, in-progress replay, secret-text non-persistence, and rollback when recovery audit insertion fails after the job mutation has begun. + +`tests/api/stripe-reconciliation-dead-letter-recovery.test.mjs` exercises unauthenticated/ordinary-member denial, owner inspection, foreign-tenant non-disclosure, bounded evidence input, one real application-route recovery against a deterministic Stripe transport seam, same-request idempotency without a second provider call, backlog clearing, and durable actor/evidence attribution. + +Both production recovery modules are part of canonical owned-production c8 instrumentation, and focused unit/API tests are part of normal repository test execution. Hosted exact-head evidence remains authoritative for integration. + +## Rollback + +Before protected integration, rollback removes the recovery route module, recovery repository/schema, bootstrap wiring, focused tests, coverage registrations, documentation, and changelog entry together. + +After integration, rollback of executable recovery behavior should stop exposing the recovery routes first but **must not delete** existing `billing_stripe_reconciliation_recoveries` rows or prior reconciliation attempts. Those rows are historical audit evidence. A later schema-retirement migration, if ever justified by retention policy, must be separately designed and reviewed. + +## Remaining work + +This slice does not add a perpetual scheduler, automatic operator paging, UI recovery console, final out-of-order end-to-end convergence acceptance, retention/export policy, or release acceptance. It also does not change `orgs.plan`, membership/RBAC, authentication semantics, or browser capability issuance. + +## References + +SQLite. (n.d.). *SAVEPOINT*. SQLite documentation. Retrieved August 21, 2026, from https://sqlite.org/lang_savepoint.html + +Stripe. (n.d.). *Receive Stripe events in your webhook endpoint*. Stripe documentation. Retrieved August 21, 2026, from https://docs.stripe.com/webhooks diff --git a/package.json b/package.json index 15937831..ea52a6cc 100644 --- a/package.json +++ b/package.json @@ -12,10 +12,10 @@ "check:python-docstrings": "node scripts/ci/static_coverage_evidence.mjs docstrings", "coverage": "npm run test:coverage", "server": "node server/server.mjs", - "test:api": "node tests/api/auth-secret.test.mjs && node --env-file=tests/api/smoke.env tests/api/smoke.mjs && node tests/api/ratelimit.test.mjs && node tests/api/attachment-status.test.mjs && node tests/api/session-revocation.test.mjs && node tests/api/orchestrator-attribution.test.mjs && node tests/api/billing-checkout.test.mjs && node tests/api/billing-live-checkout.test.mjs && node tests/api/billing-effective-plan-status.test.mjs && node tests/api/stripe-webhook.test.mjs", - "test:unit": "node tests/unit/opencode-config.test.mjs && node tests/unit/changelog-release-notes.test.mjs && node tests/unit/analytics.test.mjs && node tests/unit/cpm.test.mjs && node tests/unit/baseline-compare.test.mjs && node tests/unit/workload.test.mjs && node tests/unit/cost-evm.test.mjs && node tests/unit/msproject.test.mjs && node tests/unit/auth-password.test.mjs && node tests/unit/editor-unsaved.test.mjs && node tests/unit/static-coverage-evidence.test.mjs && node tests/unit/dep-types.test.mjs && node tests/unit/weekly-report.test.mjs && node tests/unit/clearfolio.test.mjs && node tests/unit/clearfolio-adapter-mock-hmac.test.mjs && node tests/unit/orchestrator.test.mjs && node tests/unit/orchestrator-coverage.test.mjs && node tests/unit/orchestrator-attribution.test.mjs && node tests/unit/sprint-stats.test.mjs && node tests/unit/burndown.test.mjs && node tests/unit/pm-analysis.test.mjs && node tests/unit/cloud-sync-security.test.mjs && node tests/unit/attachment-status.test.mjs && node tests/unit/clearfolio-status-signal.test.mjs && node tests/unit/coverage-script-contract.test.mjs && node tests/unit/billing-configuration.test.mjs && node tests/unit/billing-effective-plan.test.mjs && node tests/unit/billing-effective-plan-bootstrap.test.mjs && node tests/unit/billing-status-package-contract.test.mjs && node tests/unit/billing-status-response.test.mjs && node tests/unit/billing-checkout-attempt.test.mjs && node tests/unit/billing-checkout-attempt-authority.test.mjs && node tests/unit/billing-checkout-reconciliation.test.mjs && node tests/unit/billing-checkout-reconciliation-authority.test.mjs && node tests/unit/billing-checkout.test.mjs && node tests/unit/billing-provider-boundary.test.mjs && node tests/unit/billing-checkout-review-regressions.test.mjs && node tests/unit/stripe-webhook-boundary.test.mjs && node tests/unit/stripe-webhook-event-ledger.test.mjs && node tests/unit/stripe-webhook-recorder-integration.test.mjs && node tests/unit/stripe-subscription-provider.test.mjs && node tests/unit/stripe-subscription-metadata-propagation.test.mjs && node tests/unit/stripe-subscription-observation-ledger.test.mjs && node tests/unit/stripe-subscription-current-projection.test.mjs && node tests/unit/stripe-entitlement-policy.test.mjs && node tests/unit/stripe-entitlement-policy-edge.test.mjs && node tests/unit/stripe-entitlement-policy-duplicate-claims.test.mjs && node tests/unit/stripe-invoice-provider.test.mjs && node tests/unit/stripe-invoice-provider-edge.test.mjs && node tests/unit/stripe-invoice-observation-ledger.test.mjs && node tests/unit/stripe-invoice-current-projection-package-contract.test.mjs && node tests/unit/stripe-invoice-current-projection.test.mjs && node tests/unit/stripe-entitlement-claim-package-contract.test.mjs && node tests/unit/stripe-entitlement-claim-ledger.test.mjs && node tests/unit/stripe-billing-authoritative-reconciliation.test.mjs && node tests/unit/stripe-webhook-reconciliation-queue.test.mjs && node tests/unit/stripe-checkout-identity-bootstrap-package-contract.test.mjs && node tests/unit/stripe-checkout-identity-bootstrap.test.mjs && node tests/unit/stripe-webhook-reconciliation-queue-integration.test.mjs && node tests/unit/stripe-reconciliation-worker.test.mjs && node tests/unit/stripe-reconciliation-worker-time-budget.test.mjs && node tests/unit/toast-accessibility.test.mjs", - "test:coverage": "c8 --all --include=app.js --include=cloud-sync.js --include=scripts/ci/static_coverage_evidence.mjs --include=server/attachment_status.mjs --include=server/app.mjs --include=server/application_routes.mjs --include=server/auth.mjs --include=server/billing.mjs --include=server/billing_checkout_attempt.mjs --include=server/billing_configuration.mjs --include=server/billing_status_response.mjs --include=server/clearfolio.mjs --include=server/orchestrator.mjs --include=server/stripe_webhook.mjs --include=server/stripe_webhook_event_ledger.mjs --include=server/stripe_subscription_provider.mjs --include=server/stripe_subscription_observation_ledger.mjs --include=server/stripe_subscription_current_projection.mjs --include=server/stripe_entitlement_policy.mjs --include=server/stripe_invoice_provider.mjs --include=server/stripe_invoice_observation_ledger.mjs --include=server/stripe_invoice_current_projection.mjs --include=server/stripe_entitlement_claim_ledger.mjs --include=server/stripe_billing_reconciliation.mjs --include=server/stripe_webhook_reconciliation_queue.mjs --include=server/stripe_checkout_identity_bootstrap.mjs --include=server/stripe_reconciliation_worker.mjs --reporter=json --reporter=json-summary npm run test:coverage:cases", - "test:coverage:cases": "node tests/unit/coverage-script-contract.test.mjs && node tests/unit/attachment-status.test.mjs && node tests/unit/clearfolio-status-signal.test.mjs && node tests/unit/clearfolio-adapter-mock-hmac.test.mjs && node tests/unit/orchestrator.test.mjs && node tests/unit/orchestrator-coverage.test.mjs && node tests/unit/orchestrator-attribution.test.mjs && node tests/unit/msproject.test.mjs && node tests/unit/auth-password.test.mjs && node tests/unit/editor-unsaved.test.mjs && node tests/unit/static-coverage-evidence.test.mjs && node tests/unit/billing-configuration.test.mjs && node tests/unit/billing-effective-plan.test.mjs && node tests/unit/billing-effective-plan-bootstrap.test.mjs && node tests/unit/billing-status-package-contract.test.mjs && node tests/unit/billing-status-response.test.mjs && node tests/unit/billing-checkout-attempt.test.mjs && node tests/unit/billing-checkout-attempt-authority.test.mjs && node tests/unit/billing-checkout-reconciliation.test.mjs && node tests/unit/billing-checkout-reconciliation-authority.test.mjs && node tests/unit/billing-checkout.test.mjs && node tests/unit/billing-provider-boundary.test.mjs && node tests/unit/billing-checkout-review-regressions.test.mjs && node tests/unit/stripe-webhook-boundary.test.mjs && node tests/unit/stripe-webhook-event-ledger.test.mjs && node tests/unit/stripe-webhook-recorder-integration.test.mjs && node tests/unit/stripe-subscription-provider.test.mjs && node tests/unit/stripe-subscription-metadata-propagation.test.mjs && node tests/unit/stripe-subscription-observation-ledger.test.mjs && node tests/unit/stripe-subscription-current-projection.test.mjs && node tests/unit/stripe-entitlement-policy.test.mjs && node tests/unit/stripe-entitlement-policy-edge.test.mjs && node tests/unit/stripe-entitlement-policy-duplicate-claims.test.mjs && node tests/unit/stripe-invoice-provider.test.mjs && node tests/unit/stripe-invoice-provider-edge.test.mjs && node tests/unit/stripe-invoice-observation-ledger.test.mjs && node tests/unit/stripe-invoice-current-projection-package-contract.test.mjs && node tests/unit/stripe-invoice-current-projection.test.mjs && node tests/unit/stripe-entitlement-claim-package-contract.test.mjs && node tests/unit/stripe-entitlement-claim-ledger.test.mjs && node tests/unit/stripe-billing-authoritative-reconciliation.test.mjs && node tests/unit/stripe-webhook-reconciliation-queue.test.mjs && node tests/unit/stripe-checkout-identity-bootstrap.test.mjs && node tests/unit/stripe-webhook-reconciliation-queue-integration.test.mjs && node tests/unit/stripe-reconciliation-worker.test.mjs && node tests/unit/stripe-reconciliation-worker-time-budget.test.mjs && npm run test:api", + "test:api": "node tests/api/auth-secret.test.mjs && node --env-file=tests/api/smoke.env tests/api/smoke.mjs && node tests/api/ratelimit.test.mjs && node tests/api/attachment-status.test.mjs && node tests/api/session-revocation.test.mjs && node tests/api/orchestrator-attribution.test.mjs && node tests/api/billing-checkout.test.mjs && node tests/api/billing-live-checkout.test.mjs && node tests/api/billing-effective-plan-status.test.mjs && node tests/api/stripe-webhook.test.mjs && node tests/api/stripe-reconciliation-dead-letter-recovery.test.mjs", + "test:unit": "node tests/unit/opencode-config.test.mjs && node tests/unit/changelog-release-notes.test.mjs && node tests/unit/analytics.test.mjs && node tests/unit/cpm.test.mjs && node tests/unit/baseline-compare.test.mjs && node tests/unit/workload.test.mjs && node tests/unit/cost-evm.test.mjs && node tests/unit/msproject.test.mjs && node tests/unit/auth-password.test.mjs && node tests/unit/editor-unsaved.test.mjs && node tests/unit/static-coverage-evidence.test.mjs && node tests/unit/dep-types.test.mjs && node tests/unit/weekly-report.test.mjs && node tests/unit/clearfolio.test.mjs && node tests/unit/clearfolio-adapter-mock-hmac.test.mjs && node tests/unit/orchestrator.test.mjs && node tests/unit/orchestrator-coverage.test.mjs && node tests/unit/orchestrator-attribution.test.mjs && node tests/unit/sprint-stats.test.mjs && node tests/unit/burndown.test.mjs && node tests/unit/pm-analysis.test.mjs && node tests/unit/cloud-sync-security.test.mjs && node tests/unit/attachment-status.test.mjs && node tests/unit/clearfolio-status-signal.test.mjs && node tests/unit/coverage-script-contract.test.mjs && node tests/unit/billing-configuration.test.mjs && node tests/unit/billing-effective-plan.test.mjs && node tests/unit/billing-effective-plan-bootstrap.test.mjs && node tests/unit/billing-status-package-contract.test.mjs && node tests/unit/billing-status-response.test.mjs && node tests/unit/billing-checkout-attempt.test.mjs && node tests/unit/billing-checkout-attempt-authority.test.mjs && node tests/unit/billing-checkout-reconciliation.test.mjs && node tests/unit/billing-checkout-reconciliation-authority.test.mjs && node tests/unit/billing-checkout.test.mjs && node tests/unit/billing-provider-boundary.test.mjs && node tests/unit/billing-checkout-review-regressions.test.mjs && node tests/unit/stripe-webhook-boundary.test.mjs && node tests/unit/stripe-webhook-event-ledger.test.mjs && node tests/unit/stripe-webhook-recorder-integration.test.mjs && node tests/unit/stripe-subscription-provider.test.mjs && node tests/unit/stripe-subscription-metadata-propagation.test.mjs && node tests/unit/stripe-subscription-observation-ledger.test.mjs && node tests/unit/stripe-subscription-current-projection.test.mjs && node tests/unit/stripe-entitlement-policy.test.mjs && node tests/unit/stripe-entitlement-policy-edge.test.mjs && node tests/unit/stripe-entitlement-policy-duplicate-claims.test.mjs && node tests/unit/stripe-invoice-provider.test.mjs && node tests/unit/stripe-invoice-provider-edge.test.mjs && node tests/unit/stripe-invoice-observation-ledger.test.mjs && node tests/unit/stripe-invoice-current-projection-package-contract.test.mjs && node tests/unit/stripe-invoice-current-projection.test.mjs && node tests/unit/stripe-entitlement-claim-package-contract.test.mjs && node tests/unit/stripe-entitlement-claim-ledger.test.mjs && node tests/unit/stripe-billing-authoritative-reconciliation.test.mjs && node tests/unit/stripe-webhook-reconciliation-queue.test.mjs && node tests/unit/stripe-checkout-identity-bootstrap-package-contract.test.mjs && node tests/unit/stripe-checkout-identity-bootstrap.test.mjs && node tests/unit/stripe-webhook-reconciliation-queue-integration.test.mjs && node tests/unit/stripe-reconciliation-worker.test.mjs && node tests/unit/stripe-reconciliation-worker-time-budget.test.mjs && node tests/unit/stripe-reconciliation-dead-letter-recovery.test.mjs && node tests/fuzz/stripeReconciliationRecovery.fuzz.mjs && node tests/unit/toast-accessibility.test.mjs", + "test:coverage": "c8 --all --include=app.js --include=cloud-sync.js --include=scripts/ci/static_coverage_evidence.mjs --include=server/attachment_status.mjs --include=server/app.mjs --include=server/application_routes.mjs --include=server/auth.mjs --include=server/billing.mjs --include=server/billing_checkout_attempt.mjs --include=server/billing_configuration.mjs --include=server/billing_status_response.mjs --include=server/clearfolio.mjs --include=server/orchestrator.mjs --include=server/stripe_webhook.mjs --include=server/stripe_webhook_event_ledger.mjs --include=server/stripe_subscription_provider.mjs --include=server/stripe_subscription_observation_ledger.mjs --include=server/stripe_subscription_current_projection.mjs --include=server/stripe_entitlement_policy.mjs --include=server/stripe_invoice_provider.mjs --include=server/stripe_invoice_observation_ledger.mjs --include=server/stripe_invoice_current_projection.mjs --include=server/stripe_entitlement_claim_ledger.mjs --include=server/stripe_billing_reconciliation.mjs --include=server/stripe_webhook_reconciliation_queue.mjs --include=server/stripe_checkout_identity_bootstrap.mjs --include=server/stripe_reconciliation_worker.mjs --include=server/stripe_reconciliation_recovery.mjs --include=server/stripe_reconciliation_recovery_routes.mjs --reporter=json --reporter=json-summary npm run test:coverage:cases", + "test:coverage:cases": "node tests/unit/coverage-script-contract.test.mjs && node tests/unit/attachment-status.test.mjs && node tests/unit/clearfolio-status-signal.test.mjs && node tests/unit/clearfolio-adapter-mock-hmac.test.mjs && node tests/unit/orchestrator.test.mjs && node tests/unit/orchestrator-coverage.test.mjs && node tests/unit/orchestrator-attribution.test.mjs && node tests/unit/msproject.test.mjs && node tests/unit/auth-password.test.mjs && node tests/unit/editor-unsaved.test.mjs && node tests/unit/static-coverage-evidence.test.mjs && node tests/unit/billing-configuration.test.mjs && node tests/unit/billing-effective-plan.test.mjs && node tests/unit/billing-effective-plan-bootstrap.test.mjs && node tests/unit/billing-status-package-contract.test.mjs && node tests/unit/billing-status-response.test.mjs && node tests/unit/billing-checkout-attempt.test.mjs && node tests/unit/billing-checkout-attempt-authority.test.mjs && node tests/unit/billing-checkout-reconciliation.test.mjs && node tests/unit/billing-checkout-reconciliation-authority.test.mjs && node tests/unit/billing-checkout.test.mjs && node tests/unit/billing-provider-boundary.test.mjs && node tests/unit/billing-checkout-review-regressions.test.mjs && node tests/unit/stripe-webhook-boundary.test.mjs && node tests/unit/stripe-webhook-event-ledger.test.mjs && node tests/unit/stripe-webhook-recorder-integration.test.mjs && node tests/unit/stripe-subscription-provider.test.mjs && node tests/unit/stripe-subscription-metadata-propagation.test.mjs && node tests/unit/stripe-subscription-observation-ledger.test.mjs && node tests/unit/stripe-subscription-current-projection.test.mjs && node tests/unit/stripe-entitlement-policy.test.mjs && node tests/unit/stripe-entitlement-policy-edge.test.mjs && node tests/unit/stripe-entitlement-policy-duplicate-claims.test.mjs && node tests/unit/stripe-invoice-provider.test.mjs && node tests/unit/stripe-invoice-provider-edge.test.mjs && node tests/unit/stripe-invoice-observation-ledger.test.mjs && node tests/unit/stripe-invoice-current-projection-package-contract.test.mjs && node tests/unit/stripe-invoice-current-projection.test.mjs && node tests/unit/stripe-entitlement-claim-package-contract.test.mjs && node tests/unit/stripe-entitlement-claim-ledger.test.mjs && node tests/unit/stripe-billing-authoritative-reconciliation.test.mjs && node tests/unit/stripe-webhook-reconciliation-queue.test.mjs && node tests/unit/stripe-checkout-identity-bootstrap.test.mjs && node tests/unit/stripe-webhook-reconciliation-queue-integration.test.mjs && node tests/unit/stripe-reconciliation-worker.test.mjs && node tests/unit/stripe-reconciliation-worker-time-budget.test.mjs && node tests/unit/stripe-reconciliation-dead-letter-recovery.test.mjs && node tests/fuzz/stripeReconciliationRecovery.fuzz.mjs && npm run test:api", "test:e2e": "playwright test", "test:e2e:headed": "playwright test --headed", "test:e2e:cloud": "playwright install chromium && playwright test tests/e2e/cloud.spec.js tests/e2e/toast-accessibility.spec.js", diff --git a/server/app.mjs b/server/app.mjs index 8bae5fee..82e9c63e 100644 --- a/server/app.mjs +++ b/server/app.mjs @@ -4,6 +4,7 @@ import { app as applicationRoutes } from './application_routes.mjs'; import { configureBillingEntitlementDatabase } from './billing.mjs'; import { normalizeBillingStatusResponse } from './billing_status_response.mjs'; import { db } from './db.mjs'; +import { stripeReconciliationRecoveryRoutes } from './stripe_reconciliation_recovery_routes.mjs'; const toastStylesheetUrl = new URL('../toast-state.css', import.meta.url); @@ -55,4 +56,8 @@ app.use('/api/orgs/:id/billing', async (c, next) => { }); }); +// Keep operator recovery isolated from the legacy monolith. This dedicated route +// module owns only tenant-scoped dead-letter inspection/retry and is mounted before +// the broader application graph so it cannot be shadowed by future catch-all routes. +app.route('/', stripeReconciliationRecoveryRoutes); app.route('/', applicationRoutes); diff --git a/server/db.mjs b/server/db.mjs index bf18db64..4f250a4e 100644 --- a/server/db.mjs +++ b/server/db.mjs @@ -14,6 +14,11 @@ import { installStripeReconciliationWorkerSchema, runNextStripeReconciliationJob, } from './stripe_reconciliation_worker.mjs'; +import { + createSqliteStripeReconciliationRecoveryRepository, + installStripeReconciliationRecoverySchema, + retryStripeReconciliationDeadLetter, +} from './stripe_reconciliation_recovery.mjs'; import { configureStripeWebhookEventRecorder, createSqliteStripeWebhookEventRepository, @@ -263,6 +268,8 @@ export const stripeEntitlementClaims = createSqliteStripeEntitlementClaimReposit }); installStripeReconciliationWorkerSchema(db); export const stripeReconciliationWorker = createSqliteStripeReconciliationWorkerRepository(db); +installStripeReconciliationRecoverySchema(db); +export const stripeReconciliationRecoveries = createSqliteStripeReconciliationRecoveryRepository(db); /** * Consume at most one pending verified Stripe reconciliation trigger. @@ -285,5 +292,39 @@ export function reconcileNextStripeBillingTrigger() { }); } +/** + * Retry one exact tenant-owned Stripe reconciliation dead letter. + * + * Authenticated adapters provide only the tenant, verified Event identity, actor, and + * evidence reference. The recovery repository resolves Subscription authority from + * normalized server state and the reconciliation service re-fetches current provider + * state before entitlement evaluation. + * + * @param {{organizationId:number,eventId:string,actorUserId:number,evidenceReference:string}} input + * operator recovery authority + * @returns {Promise>} durable recovery receipt + */ +export function recoverStripeBillingDeadLetter({ + organizationId, + eventId, + actorUserId, + evidenceReference, +}) { + return retryStripeReconciliationDeadLetter({ + recoveryRepository: stripeReconciliationRecoveries, + workerRepository: stripeReconciliationWorker, + reconcile: reconcileStripeBillingAuthoritatively, + reconciliationDependencies: { + subscriptionRepository: stripeSubscriptionObservations, + invoiceRepository: stripeInvoiceObservations, + claimRepository: stripeEntitlementClaims, + }, + organizationId, + eventId, + actorUserId, + evidenceReference, + }); +} + // node:sqlite returns lastInsertRowid as number|bigint; normalize to Number. export const rowid = (r) => Number(r.lastInsertRowid); diff --git a/server/stripe_reconciliation_recovery.mjs b/server/stripe_reconciliation_recovery.mjs new file mode 100644 index 00000000..e5379faa --- /dev/null +++ b/server/stripe_reconciliation_recovery.mjs @@ -0,0 +1,715 @@ +import { createHash, randomUUID } from 'node:crypto'; + +const MAX_PROVIDER_ID_LENGTH = 255; +const MAX_EVIDENCE_REFERENCE_LENGTH = 256; +const MAX_LIST_LIMIT = 100; +const DEFAULT_LIST_LIMIT = 50; +const DEFAULT_LEASE_MS = 90_000; +const LEASE_EXPIRED_CODE = 'stripe_reconciliation_lease_expired'; +const EVENT_ID_PATTERN = /^[A-Za-z0-9_:-]+$/u; +const SUBSCRIPTION_ID_PATTERN = /^sub_[A-Za-z0-9_]+$/u; +const LEASE_TOKEN_PATTERN = /^[A-Za-z0-9_-]{16,128}$/u; +const ERROR_CODE_PATTERN = /^[a-z0-9_:-]+$/u; +const CONTROL_CHARACTER_PATTERN = /[\u0000-\u001f\u007f]/u; +const SAVEPOINT_NAME = 'billing_stripe_reconciliation_recovery_write'; + +/** Stable fail-closed error for operator-initiated Stripe reconciliation recovery. */ +export class StripeReconciliationRecoveryError extends Error { + /** + * Create one sanitized recovery failure. + * @param {string} code stable machine-readable failure code + * @param {number} [status=400] HTTP-compatible status for an API adapter + */ + constructor(code, status = 400) { + super(code); + this.name = 'StripeReconciliationRecoveryError'; + this.code = code; + this.status = status; + } +} + +function recoveryError(code, status = 400) { + return new StripeReconciliationRecoveryError(code, status); +} + +function positiveInteger(value) { + if (!Number.isSafeInteger(value) || value <= 0) { + throw recoveryError('stripe_reconciliation_recovery_invalid'); + } + return value; +} + +function boundedIdentifier(value, pattern) { + if ( + typeof value !== 'string' + || value.length === 0 + || value.length > MAX_PROVIDER_ID_LENGTH + || !pattern.test(value) + ) { + throw recoveryError('stripe_reconciliation_recovery_invalid'); + } + return value; +} + +function evidenceReferenceValue(value) { + if (typeof value !== 'string') { + throw recoveryError('stripe_reconciliation_recovery_invalid'); + } + const normalized = value.trim(); + if ( + normalized.length === 0 + || normalized.length > MAX_EVIDENCE_REFERENCE_LENGTH + || CONTROL_CHARACTER_PATTERN.test(normalized) + ) { + throw recoveryError('stripe_reconciliation_recovery_invalid'); + } + return normalized; +} + +function listLimitValue(value) { + if (value === undefined) return DEFAULT_LIST_LIMIT; + if (!Number.isSafeInteger(value) || value <= 0 || value > MAX_LIST_LIMIT) { + throw recoveryError('stripe_reconciliation_recovery_invalid'); + } + return value; +} + +function normalizedNow(now) { + const value = Number(now()); + if (!Number.isSafeInteger(value) || value < 0) { + throw recoveryError('stripe_reconciliation_recovery_clock_invalid', 500); + } + return value; +} + +function leaseTokenValue(randomToken) { + const value = randomToken(); + if (typeof value !== 'string' || !LEASE_TOKEN_PATTERN.test(value)) { + throw recoveryError('stripe_reconciliation_recovery_token_invalid', 500); + } + return value; +} + +function tokenHash(value) { + return createHash('sha256').update(value, 'utf8').digest('hex'); +} + +function safeAdd(left, right) { + const value = left + right; + if (!Number.isSafeInteger(left) || !Number.isSafeInteger(right) + || left < 0 || right <= 0 || !Number.isSafeInteger(value)) { + throw recoveryError('stripe_reconciliation_recovery_clock_invalid', 500); + } + return value; +} + +function safeFailureCode(error) { + const code = error && typeof error === 'object' ? error.code : null; + if ( + typeof code === 'string' + && code.startsWith('stripe_') + && code.length <= 96 + && ERROR_CODE_PATTERN.test(code) + ) { + return code; + } + return 'stripe_reconciliation_recovery_failed'; +} + +function withSavepoint(database, operation) { + database.exec(`SAVEPOINT ${SAVEPOINT_NAME}`); + try { + const result = operation(); + database.exec(`RELEASE SAVEPOINT ${SAVEPOINT_NAME}`); + return result; + } catch (error) { + let rolledBack = false; + try { + database.exec(`ROLLBACK TO SAVEPOINT ${SAVEPOINT_NAME}`); + rolledBack = true; + } catch { + // Leave an unconfirmed failed savepoint open instead of risking partial commit. + } + if (rolledBack) { + try { + database.exec(`RELEASE SAVEPOINT ${SAVEPOINT_NAME}`); + } catch { + // Cleanup after a confirmed rollback must not replace the causal failure. + } + } + throw error; + } +} + +function replayReceipt(row) { + if (!row) return null; + const base = { + recoveryId: positiveInteger(Number(row.recovery_id)), + eventId: boundedIdentifier(row.event_id, EVENT_ID_PATTERN), + subscriptionId: boundedIdentifier(row.subscription_id, SUBSCRIPTION_ID_PATTERN), + attemptNumber: positiveInteger(Number(row.attempt_number)), + }; + const effectiveOutcome = row.recovery_outcome ?? row.attempt_outcome; + if (effectiveOutcome == null) { + return Object.freeze({ status: 'processing', replayed: true, ...base }); + } + if (effectiveOutcome === 'succeeded') { + return Object.freeze({ + status: 'succeeded', + replayed: true, + ...base, + claimDecisionId: positiveInteger(Number(row.recovery_claim_decision_id ?? row.job_claim_decision_id)), + }); + } + if (effectiveOutcome === 'dead_letter') { + const errorCode = row.recovery_error_code ?? row.attempt_error_code; + if (typeof errorCode !== 'string' || !ERROR_CODE_PATTERN.test(errorCode)) { + throw recoveryError('stripe_reconciliation_recovery_state_invalid', 500); + } + return Object.freeze({ status: 'dead_letter', replayed: true, ...base, errorCode }); + } + throw recoveryError('stripe_reconciliation_recovery_state_invalid', 500); +} + +/** + * Install immutable operator-recovery evidence for Stripe reconciliation dead letters. + * + * Recovery authority is normalized: tenant identity stays on the existing + * Subscription→Customer chain, attempt outcome stays on worker attempt history, and + * this table records only who authorized which exact manual attempt and why. The + * evidence reference is idempotency authority for one event and is never a provider + * credential or browser capability. + * + * @param {import('node:sqlite').DatabaseSync} database bootstrapped SQLite database + * @returns {void} + */ +export function installStripeReconciliationRecoverySchema(database) { + if (!database || typeof database.exec !== 'function') { + throw new TypeError('database must provide SQLite exec operations'); + } + database.exec(` + CREATE TABLE IF NOT EXISTS billing_stripe_reconciliation_recoveries ( + recovery_id INTEGER PRIMARY KEY AUTOINCREMENT, + event_id TEXT NOT NULL, + attempt_number INTEGER NOT NULL CHECK(attempt_number > 0), + actor_user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE RESTRICT, + evidence_reference TEXT NOT NULL + CHECK(length(evidence_reference) BETWEEN 1 AND ${MAX_EVIDENCE_REFERENCE_LENGTH}), + requested_at_ms INTEGER NOT NULL CHECK(requested_at_ms >= 0), + completed_at_ms INTEGER CHECK(completed_at_ms IS NULL OR completed_at_ms >= requested_at_ms), + outcome TEXT CHECK(outcome IS NULL OR outcome IN ('succeeded','dead_letter')), + error_code TEXT CHECK(error_code IS NULL OR length(error_code) BETWEEN 1 AND 96), + claim_decision_id INTEGER CHECK(claim_decision_id IS NULL OR claim_decision_id > 0), + FOREIGN KEY(event_id, attempt_number) + REFERENCES billing_stripe_reconciliation_attempts(event_id, attempt_number) + ON DELETE RESTRICT, + UNIQUE(event_id, evidence_reference), + CHECK( + (outcome IS NULL AND completed_at_ms IS NULL + AND error_code IS NULL AND claim_decision_id IS NULL) + OR + (outcome = 'succeeded' AND completed_at_ms IS NOT NULL + AND error_code IS NULL AND claim_decision_id IS NOT NULL) + OR + (outcome = 'dead_letter' AND completed_at_ms IS NOT NULL + AND error_code IS NOT NULL AND claim_decision_id IS NULL) + ) + ); + + CREATE INDEX IF NOT EXISTS billing_stripe_reconciliation_recovery_actor_history + ON billing_stripe_reconciliation_recoveries(actor_user_id, requested_at_ms, recovery_id); + `); +} + +/** + * Create the tenant-scoped repository used by operator dead-letter recovery. + * + * A manual recovery never resets automatic attempt history. It adds exactly one new + * leased attempt after a dead letter, preserving every prior attempt. Reusing the + * same event/evidence-reference pair returns the existing recovery receipt without + * creating another provider call or attempt. A fresh explicit evidence reference is + * required for another manual attempt. + * + * @param {import('node:sqlite').DatabaseSync} database bootstrapped SQLite database + * @param {object} [options] deterministic runtime controls + * @param {() => number} [options.now] wall-clock milliseconds + * @param {() => string} [options.randomToken] opaque lease-token generator + * @param {number} [options.leaseMs=90000] manual recovery lease lifetime + * @returns {Readonly} recovery repository + */ +export function createSqliteStripeReconciliationRecoveryRepository(database, { + now = Date.now, + randomToken = randomUUID, + leaseMs = DEFAULT_LEASE_MS, +} = {}) { + if (!database || typeof database.prepare !== 'function' || typeof database.exec !== 'function') { + throw new TypeError('database must provide SQLite prepare/exec operations'); + } + if (typeof now !== 'function') throw new TypeError('now must be a function'); + if (typeof randomToken !== 'function') throw new TypeError('randomToken must be a function'); + if (!Number.isSafeInteger(leaseMs) || leaseMs <= 0) { + throw new TypeError('leaseMs must be a positive safe integer'); + } + + const listDeadLettersQuery = database.prepare(` + SELECT jobs.event_id, triggers.subscription_id, jobs.attempt_count, + jobs.completed_at_ms, jobs.last_error_code + FROM billing_stripe_reconciliation_jobs AS jobs + JOIN billing_stripe_reconciliation_triggers AS triggers USING(event_id) + JOIN billing_stripe_subscriptions AS subscriptions + ON subscriptions.subscription_id = triggers.subscription_id + JOIN billing_stripe_customers AS customers + ON customers.customer_id = subscriptions.customer_id + WHERE customers.organization_id = ? + AND jobs.processing_state = 'dead_letter' + ORDER BY jobs.completed_at_ms DESC, jobs.event_id + LIMIT ? + `); + const selectDeadLetter = database.prepare(` + SELECT jobs.event_id, triggers.subscription_id, jobs.attempt_count + FROM billing_stripe_reconciliation_jobs AS jobs + JOIN billing_stripe_reconciliation_triggers AS triggers USING(event_id) + JOIN billing_stripe_subscriptions AS subscriptions + ON subscriptions.subscription_id = triggers.subscription_id + JOIN billing_stripe_customers AS customers + ON customers.customer_id = subscriptions.customer_id + WHERE customers.organization_id = ? + AND jobs.event_id = ? + AND jobs.processing_state = 'dead_letter' + `); + const selectExistingRecovery = database.prepare(` + SELECT recoveries.recovery_id, recoveries.event_id, recoveries.attempt_number, + recoveries.outcome AS recovery_outcome, + recoveries.error_code AS recovery_error_code, + recoveries.claim_decision_id AS recovery_claim_decision_id, + attempts.outcome AS attempt_outcome, + attempts.error_code AS attempt_error_code, + jobs.claim_decision_id AS job_claim_decision_id, + triggers.subscription_id + FROM billing_stripe_reconciliation_recoveries AS recoveries + JOIN billing_stripe_reconciliation_attempts AS attempts + ON attempts.event_id = recoveries.event_id + AND attempts.attempt_number = recoveries.attempt_number + JOIN billing_stripe_reconciliation_jobs AS jobs + ON jobs.event_id = recoveries.event_id + JOIN billing_stripe_reconciliation_triggers AS triggers + ON triggers.event_id = recoveries.event_id + JOIN billing_stripe_subscriptions AS subscriptions + ON subscriptions.subscription_id = triggers.subscription_id + JOIN billing_stripe_customers AS customers + ON customers.customer_id = subscriptions.customer_id + WHERE customers.organization_id = ? + AND recoveries.event_id = ? + AND recoveries.evidence_reference = ? + `); + const selectActiveRecovery = database.prepare(` + SELECT recoveries.recovery_id, recoveries.event_id, recoveries.attempt_number, + jobs.attempt_count AS job_attempt_count, + jobs.lease_expires_at_ms AS job_lease_expires_at_ms + FROM billing_stripe_reconciliation_recoveries AS recoveries + JOIN billing_stripe_reconciliation_attempts AS attempts + ON attempts.event_id = recoveries.event_id + AND attempts.attempt_number = recoveries.attempt_number + JOIN billing_stripe_reconciliation_jobs AS jobs + ON jobs.event_id = recoveries.event_id + JOIN billing_stripe_reconciliation_triggers AS triggers + ON triggers.event_id = recoveries.event_id + JOIN billing_stripe_subscriptions AS subscriptions + ON subscriptions.subscription_id = triggers.subscription_id + JOIN billing_stripe_customers AS customers + ON customers.customer_id = subscriptions.customer_id + WHERE customers.organization_id = ? + AND recoveries.event_id = ? + AND recoveries.outcome IS NULL + AND attempts.outcome IS NULL + AND jobs.processing_state = 'processing' + AND jobs.attempt_count = recoveries.attempt_number + ORDER BY recoveries.recovery_id DESC + LIMIT 1 + `); + const claimJob = database.prepare(` + UPDATE billing_stripe_reconciliation_jobs + SET processing_state = 'processing', + attempt_count = attempt_count + 1, + next_attempt_at_ms = ?, + lease_token_sha256 = ?, + lease_expires_at_ms = ?, + completed_at_ms = NULL, + last_error_code = NULL, + claim_decision_id = NULL + WHERE event_id = ? + AND processing_state = 'dead_letter' + AND attempt_count = ? + `); + const insertAttempt = database.prepare(` + INSERT INTO billing_stripe_reconciliation_attempts( + event_id, attempt_number, lease_started_at_ms, lease_expires_at_ms, + finished_at_ms, outcome, error_code + ) VALUES(?,?,?,?,NULL,NULL,NULL) + `); + const insertRecovery = database.prepare(` + INSERT INTO billing_stripe_reconciliation_recoveries( + event_id, attempt_number, actor_user_id, evidence_reference, + requested_at_ms, completed_at_ms, outcome, error_code, claim_decision_id + ) VALUES(?,?,?,?,?,NULL,NULL,NULL,NULL) + `); + const finishRecoverySuccess = database.prepare(` + UPDATE billing_stripe_reconciliation_recoveries + SET completed_at_ms = ?, outcome = 'succeeded', error_code = NULL, + claim_decision_id = ? + WHERE recovery_id = ? AND event_id = ? AND attempt_number = ? AND outcome IS NULL + `); + const finishRecoveryFailure = database.prepare(` + UPDATE billing_stripe_reconciliation_recoveries + SET completed_at_ms = ?, outcome = 'dead_letter', error_code = ?, + claim_decision_id = NULL + WHERE recovery_id = ? AND event_id = ? AND attempt_number = ? AND outcome IS NULL + `); + const finishExpiredAttempt = database.prepare(` + UPDATE billing_stripe_reconciliation_attempts + SET finished_at_ms = ?, outcome = 'dead_letter', error_code = ? + WHERE event_id = ? AND attempt_number = ? AND outcome IS NULL + AND lease_expires_at_ms <= ? + `); + const finishExpiredJob = database.prepare(` + UPDATE billing_stripe_reconciliation_jobs + SET processing_state = 'dead_letter', next_attempt_at_ms = ?, + lease_token_sha256 = NULL, lease_expires_at_ms = NULL, + completed_at_ms = ?, last_error_code = ?, claim_decision_id = NULL + WHERE event_id = ? AND processing_state = 'processing' + AND attempt_count = ? AND lease_expires_at_ms <= ? + `); + + function recoveryNow() { + return normalizedNow(now); + } + + function reapExpiredRecovery(orgId, eventId, nowMs) { + const candidate = selectActiveRecovery.get(orgId, eventId); + if (!candidate) return false; + const candidateLeaseExpiresAtMs = Number(candidate.job_lease_expires_at_ms); + if (!Number.isSafeInteger(candidateLeaseExpiresAtMs) || candidateLeaseExpiresAtMs < 0) { + throw recoveryError('stripe_reconciliation_recovery_state_invalid', 500); + } + if (candidateLeaseExpiresAtMs > nowMs) return false; + + return withSavepoint(database, () => { + const current = selectActiveRecovery.get(orgId, eventId); + if (!current) return false; + const recoveryId = positiveInteger(Number(current.recovery_id)); + const attemptNumber = positiveInteger(Number(current.attempt_number)); + const jobAttemptCount = positiveInteger(Number(current.job_attempt_count)); + const leaseExpiresAtMs = Number(current.job_lease_expires_at_ms); + if ( + jobAttemptCount !== attemptNumber + || !Number.isSafeInteger(leaseExpiresAtMs) + || leaseExpiresAtMs < 0 + ) { + throw recoveryError('stripe_reconciliation_recovery_state_invalid', 500); + } + if (leaseExpiresAtMs > nowMs) return false; + + const attemptUpdated = finishExpiredAttempt.run( + nowMs, + LEASE_EXPIRED_CODE, + eventId, + attemptNumber, + nowMs, + ); + if (Number(attemptUpdated.changes) !== 1) { + throw recoveryError('stripe_reconciliation_recovery_state_uncertain', 500); + } + const jobUpdated = finishExpiredJob.run( + nowMs, + nowMs, + LEASE_EXPIRED_CODE, + eventId, + attemptNumber, + nowMs, + ); + if (Number(jobUpdated.changes) !== 1) { + throw recoveryError('stripe_reconciliation_recovery_state_uncertain', 500); + } + const recoveryUpdated = finishRecoveryFailure.run( + nowMs, + LEASE_EXPIRED_CODE, + recoveryId, + eventId, + attemptNumber, + ); + if (Number(recoveryUpdated.changes) !== 1) { + throw recoveryError('stripe_reconciliation_recovery_state_uncertain', 500); + } + return true; + }); + } + + return Object.freeze({ + /** List bounded dead-letter summaries for one exact ScopeWeave tenant. */ + listDeadLetters({ organizationId, limit } = {}) { + const orgId = positiveInteger(organizationId); + const boundedLimit = listLimitValue(limit); + return listDeadLettersQuery.all(orgId, boundedLimit).map((row) => Object.freeze({ + eventId: boundedIdentifier(row.event_id, EVENT_ID_PATTERN), + subscriptionId: boundedIdentifier(row.subscription_id, SUBSCRIPTION_ID_PATTERN), + attemptCount: positiveInteger(Number(row.attempt_count)), + completedAtMs: Number(row.completed_at_ms), + lastErrorCode: row.last_error_code, + })); + }, + + /** Claim one exact tenant-owned dead letter under new operator recovery authority. */ + claimDeadLetterRecovery({ organizationId, eventId, actorUserId, evidenceReference } = {}) { + const orgId = positiveInteger(organizationId); + const normalizedEventId = boundedIdentifier(eventId, EVENT_ID_PATTERN); + const actorId = positiveInteger(actorUserId); + const evidence = evidenceReferenceValue(evidenceReference); + const nowMs = recoveryNow(); + + reapExpiredRecovery(orgId, normalizedEventId, nowMs); + const existing = replayReceipt(selectExistingRecovery.get(orgId, normalizedEventId, evidence)); + if (existing) return existing; + + const leaseToken = leaseTokenValue(randomToken); + const leaseExpiresAtMs = safeAdd(nowMs, leaseMs); + const leaseHash = tokenHash(leaseToken); + + try { + return withSavepoint(database, () => { + const row = selectDeadLetter.get(orgId, normalizedEventId); + if (!row) { + throw recoveryError('stripe_reconciliation_dead_letter_not_found', 404); + } + const subscriptionId = boundedIdentifier(row.subscription_id, SUBSCRIPTION_ID_PATTERN); + const priorAttemptCount = positiveInteger(Number(row.attempt_count)); + const attemptNumber = priorAttemptCount + 1; + if (!Number.isSafeInteger(attemptNumber)) { + throw recoveryError('stripe_reconciliation_recovery_state_invalid', 500); + } + const claimed = claimJob.run( + nowMs, + leaseHash, + leaseExpiresAtMs, + normalizedEventId, + priorAttemptCount, + ); + if (Number(claimed.changes) !== 1) { + throw recoveryError('stripe_reconciliation_recovery_conflict', 409); + } + insertAttempt.run( + normalizedEventId, + attemptNumber, + nowMs, + leaseExpiresAtMs, + ); + const inserted = insertRecovery.run( + normalizedEventId, + attemptNumber, + actorId, + evidence, + nowMs, + ); + const recoveryId = positiveInteger(Number(inserted.lastInsertRowid)); + return Object.freeze({ + status: 'processing', + replayed: false, + recoveryId, + eventId: normalizedEventId, + subscriptionId, + organizationId: orgId, + attemptNumber, + leaseToken, + leaseExpiresAtMs, + }); + }); + } catch (error) { + const raced = replayReceipt(selectExistingRecovery.get(orgId, normalizedEventId, evidence)); + if (raced) return raced; + throw error; + } + }, + + /** Atomically complete the worker lease and its immutable recovery receipt. */ + completeRecovery({ claim, claimDecisionId, workerRepository } = {}) { + if (!claim || typeof claim !== 'object' || claim.replayed !== false) { + throw recoveryError('stripe_reconciliation_recovery_invalid'); + } + if (!workerRepository || typeof workerRepository.complete !== 'function') { + throw new TypeError('workerRepository must provide complete()'); + } + const decisionId = positiveInteger(claimDecisionId); + const nowMs = recoveryNow(); + return withSavepoint(database, () => { + workerRepository.complete({ + eventId: claim.eventId, + leaseToken: claim.leaseToken, + claimDecisionId: decisionId, + }); + const updated = finishRecoverySuccess.run( + nowMs, + decisionId, + claim.recoveryId, + claim.eventId, + claim.attemptNumber, + ); + if (Number(updated.changes) !== 1) { + throw recoveryError('stripe_reconciliation_recovery_state_uncertain', 500); + } + }); + }, + + /** Atomically dead-letter the manual worker lease and its recovery receipt. */ + failRecovery({ claim, errorCode, workerRepository } = {}) { + if (!claim || typeof claim !== 'object' || claim.replayed !== false) { + throw recoveryError('stripe_reconciliation_recovery_invalid'); + } + if (!workerRepository || typeof workerRepository.fail !== 'function') { + throw new TypeError('workerRepository must provide fail()'); + } + const code = typeof errorCode === 'string' && ERROR_CODE_PATTERN.test(errorCode) + ? errorCode + : 'stripe_reconciliation_recovery_failed'; + const nowMs = recoveryNow(); + let workerResult; + withSavepoint(database, () => { + workerResult = workerRepository.fail({ + eventId: claim.eventId, + leaseToken: claim.leaseToken, + errorCode: code, + }); + if (!workerResult || workerResult.status !== 'dead_letter') { + throw recoveryError('stripe_reconciliation_recovery_state_uncertain', 500); + } + const updated = finishRecoveryFailure.run( + nowMs, + code, + claim.recoveryId, + claim.eventId, + claim.attemptNumber, + ); + if (Number(updated.changes) !== 1) { + throw recoveryError('stripe_reconciliation_recovery_state_uncertain', 500); + } + }); + return workerResult; + }, + }); +} + +function assertReconciliationReceipt(receipt, claim) { + if (!receipt || typeof receipt !== 'object' || Array.isArray(receipt)) { + throw recoveryError('stripe_reconciliation_recovery_receipt_invalid', 500); + } + if (receipt.organizationId !== claim.organizationId + || receipt.subscriptionId !== claim.subscriptionId + || !Number.isSafeInteger(receipt.claimDecisionId) + || receipt.claimDecisionId <= 0) { + throw recoveryError('stripe_reconciliation_recovery_receipt_invalid', 500); + } + return receipt.claimDecisionId; +} + +/** + * Retry one operator-approved dead letter through current Stripe provider authority. + * + * The caller chooses only a tenant, verified Event identity, actor, and durable evidence + * reference. Tenant/Subscription authority comes from normalized server-owned state. + * Replaying the same evidence reference is side-effect free. A new explicit reference + * creates one new finite lease and attempt; failure returns immediately to dead-letter + * state instead of reopening the automatic retry budget. Once provider reconciliation + * succeeds, any completion uncertainty fails closed without starting a contradictory + * failure transition against a lease whose durable state can no longer be trusted. + * + * @param {object} input recovery and reconciliation ports + * @param {object} input.recoveryRepository tenant-scoped recovery repository + * @param {object} input.workerRepository worker completion/failure repository + * @param {Function} input.reconcile authoritative Stripe billing reconciliation function + * @param {object} [input.reconciliationDependencies] server-owned provider/persistence ports + * @param {number} input.organizationId tenant authority requested by authenticated adapter + * @param {string} input.eventId exact verified Event dead-letter identity + * @param {number} input.actorUserId authenticated operator user ID + * @param {string} input.evidenceReference bounded operator evidence/idempotency reference + * @returns {Promise>} durable replay, success, or dead-letter receipt + */ +export async function retryStripeReconciliationDeadLetter({ + recoveryRepository, + workerRepository, + reconcile, + reconciliationDependencies = {}, + organizationId, + eventId, + actorUserId, + evidenceReference, +}) { + if (!recoveryRepository + || typeof recoveryRepository.claimDeadLetterRecovery !== 'function' + || typeof recoveryRepository.completeRecovery !== 'function' + || typeof recoveryRepository.failRecovery !== 'function') { + throw new TypeError('recoveryRepository must provide recovery operations'); + } + if (!workerRepository + || typeof workerRepository.complete !== 'function' + || typeof workerRepository.fail !== 'function') { + throw new TypeError('workerRepository must provide complete()/fail()'); + } + if (typeof reconcile !== 'function') throw new TypeError('reconcile must be a function'); + if (!reconciliationDependencies || typeof reconciliationDependencies !== 'object' + || Array.isArray(reconciliationDependencies)) { + throw new TypeError('reconciliationDependencies must be an object'); + } + + const claim = recoveryRepository.claimDeadLetterRecovery({ + organizationId, + eventId, + actorUserId, + evidenceReference, + }); + if (claim.replayed) return claim; + + let claimDecisionId; + try { + const receipt = await reconcile({ + ...reconciliationDependencies, + organizationId: claim.organizationId, + subscriptionId: claim.subscriptionId, + sourceEventId: claim.eventId, + }); + claimDecisionId = assertReconciliationReceipt(receipt, claim); + } catch (error) { + const errorCode = safeFailureCode(error); + try { + recoveryRepository.failRecovery({ claim, errorCode, workerRepository }); + } catch { + throw recoveryError('stripe_reconciliation_recovery_state_uncertain', 500); + } + return Object.freeze({ + status: 'dead_letter', + replayed: false, + recoveryId: claim.recoveryId, + eventId: claim.eventId, + subscriptionId: claim.subscriptionId, + attemptNumber: claim.attemptNumber, + errorCode, + }); + } + + try { + recoveryRepository.completeRecovery({ + claim, + claimDecisionId, + workerRepository, + }); + } catch { + throw recoveryError('stripe_reconciliation_recovery_state_uncertain', 500); + } + + return Object.freeze({ + status: 'succeeded', + replayed: false, + recoveryId: claim.recoveryId, + eventId: claim.eventId, + subscriptionId: claim.subscriptionId, + attemptNumber: claim.attemptNumber, + claimDecisionId, + }); +} diff --git a/server/stripe_reconciliation_recovery_routes.mjs b/server/stripe_reconciliation_recovery_routes.mjs new file mode 100644 index 00000000..d5dd38cc --- /dev/null +++ b/server/stripe_reconciliation_recovery_routes.mjs @@ -0,0 +1,157 @@ +import { Hono } from 'hono'; +import { bodyLimit } from 'hono/body-limit'; + +import { hashApiToken, verifyToken } from './auth.mjs'; +import { + db, + recoverStripeBillingDeadLetter, + stripeReconciliationRecoveries, +} from './db.mjs'; +import { StripeReconciliationRecoveryError } from './stripe_reconciliation_recovery.mjs'; + +const MAX_RECOVERY_REQUEST_BYTES = 4 * 1024; + +function organizationRole(userId, organizationId) { + return db.prepare('SELECT role FROM memberships WHERE user_id = ? AND org_id = ?') + .get(userId, organizationId)?.role ?? null; +} + +function canManage(role) { + return role === 'owner' || role === 'admin'; +} + +async function requireRecoveryAuth(c, next) { + const header = c.req.header('authorization') || ''; + const token = header.startsWith('Bearer ') ? header.slice(7) : ''; + + if (token.startsWith('swk_')) { + const row = db.prepare('SELECT * FROM api_tokens WHERE token_hash = ?').get(hashApiToken(token)); + if (!row) return c.json({ error: 'unauthorized' }, 401); + db.prepare("UPDATE api_tokens SET last_used = datetime('now') WHERE id = ?").run(row.id); + c.set('recoveryUserId', Number(row.user_id)); + return next(); + } + + try { + const payload = verifyToken(token); + const user = db.prepare('SELECT token_version FROM users WHERE id = ?').get(payload.sub); + if (!user || (payload.tv || 0) !== user.token_version) { + return c.json({ error: 'unauthorized' }, 401); + } + c.set('recoveryUserId', Number(payload.sub)); + } catch { + return c.json({ error: 'unauthorized' }, 401); + } + await next(); +} + +function recoveryFailure(c, error) { + if (error instanceof StripeReconciliationRecoveryError) { + return c.json({ error: error.code }, error.status, { 'Cache-Control': 'no-store' }); + } + return c.json( + { error: 'stripe_reconciliation_recovery_unavailable' }, + 500, + { 'Cache-Control': 'no-store' }, + ); +} + +function auditRecovery(organizationId, actorUserId, result) { + try { + db.prepare(` + INSERT INTO audit_log(org_id,user_id,action,target_type,target_id,meta) + VALUES(?,?,?,?,?,?) + `).run( + organizationId, + actorUserId, + 'billing.reconciliation.recover', + 'stripe_event', + result.eventId, + JSON.stringify({ + recoveryId: result.recoveryId, + attemptNumber: result.attemptNumber, + status: result.status, + replayed: result.replayed, + }), + ); + } catch { + // The normalized recovery table is the durable recovery authority. This legacy + // product audit stream is additive evidence and must not make an idempotent retry + // appear to fail after provider/state work has already completed. + } +} + +/** + * Tenant-scoped operator API for Stripe reconciliation dead-letter recovery. + * + * The route graph deliberately exposes no lease token, provider secret, raw webhook + * payload, or caller-selected Subscription identity. Owners/admins can inspect their + * bounded backlog and retry one exact verified Event using a durable evidence reference. + * Recovery JSON is capped at 4 KiB by Hono's body-limit middleware, which checks both + * declared Content-Length and streamed bytes before the JSON parser can buffer an + * unbounded privileged request. + */ +export const stripeReconciliationRecoveryRoutes = new Hono(); + +stripeReconciliationRecoveryRoutes.get( + '/api/orgs/:id/billing/reconciliation/dead-letters', + requireRecoveryAuth, + (c) => { + const actorUserId = c.get('recoveryUserId'); + const organizationId = Number(c.req.param('id')); + const role = Number.isSafeInteger(organizationId) && organizationId > 0 + ? organizationRole(actorUserId, organizationId) + : null; + if (!role) return c.json({ error: 'not found' }, 404, { 'Cache-Control': 'no-store' }); + if (!canManage(role)) return c.json({ error: 'forbidden' }, 403, { 'Cache-Control': 'no-store' }); + + const rawLimit = c.req.query('limit'); + const limit = rawLimit === undefined ? undefined : Number(rawLimit); + try { + const deadLetters = stripeReconciliationRecoveries.listDeadLetters({ + organizationId, + limit, + }); + return c.json({ deadLetters }, 200, { 'Cache-Control': 'no-store' }); + } catch (error) { + return recoveryFailure(c, error); + } + }, +); + +stripeReconciliationRecoveryRoutes.post( + '/api/orgs/:id/billing/reconciliation/dead-letters/:eventId/retry', + requireRecoveryAuth, + bodyLimit({ + maxSize: MAX_RECOVERY_REQUEST_BYTES, + onError: (c) => c.json( + { error: 'stripe_reconciliation_recovery_body_too_large' }, + 413, + { 'Cache-Control': 'no-store' }, + ), + }), + async (c) => { + const actorUserId = c.get('recoveryUserId'); + const organizationId = Number(c.req.param('id')); + const role = Number.isSafeInteger(organizationId) && organizationId > 0 + ? organizationRole(actorUserId, organizationId) + : null; + if (!role) return c.json({ error: 'not found' }, 404, { 'Cache-Control': 'no-store' }); + if (!canManage(role)) return c.json({ error: 'forbidden' }, 403, { 'Cache-Control': 'no-store' }); + + const body = await c.req.json().catch(() => ({})); + try { + const result = await recoverStripeBillingDeadLetter({ + organizationId, + eventId: c.req.param('eventId'), + actorUserId, + evidenceReference: body.evidenceReference, + }); + auditRecovery(organizationId, actorUserId, result); + const status = result.status === 'processing' ? 202 : 200; + return c.json(result, status, { 'Cache-Control': 'no-store' }); + } catch (error) { + return recoveryFailure(c, error); + } + }, +); diff --git a/tests/api/stripe-reconciliation-dead-letter-recovery.test.mjs b/tests/api/stripe-reconciliation-dead-letter-recovery.test.mjs new file mode 100644 index 00000000..1d722a4e --- /dev/null +++ b/tests/api/stripe-reconciliation-dead-letter-recovery.test.mjs @@ -0,0 +1,260 @@ +import assert from 'node:assert/strict'; + +process.env.SCOPEWEAVE_DB = ':memory:'; +process.env.SCOPEWEAVE_DEV = '1'; +process.env.SCOPEWEAVE_PUBLIC_ORIGIN = 'https://scopeweave.test'; +process.env.SCOPEWEAVE_JWT_SECRET = '0123456789abcdef0123456789abcdef'; +process.env.STRIPE_SECRET_KEY = 'sk_test_dead_letter_recovery'; +process.env.STRIPE_PRICE_ID = 'price_dead_letter_recovery'; +process.env.STRIPE_WEBHOOK_SECRET = 'whsec_dead_letter_recovery'; +delete process.env.ORCHESTRATOR_URL; + +const { app } = await import('../../server/app.mjs'); +const { db } = await import('../../server/db.mjs'); + +const request = (path, options = {}) => app.request(path, { + ...options, + headers: { 'content-type': 'application/json', ...(options.headers || {}) }, +}); +const jsonBody = (value) => JSON.stringify(value); + +async function signup(email, name) { + const response = await request('/api/auth/signup', { + method: 'POST', + body: jsonBody({ email, password: 'password123', name }), + }); + assert.equal(response.status, 200, `signup succeeds for ${email}`); + const payload = await response.json(); + const me = await request('/api/me', { + headers: { authorization: `Bearer ${payload.token}` }, + }); + const identity = await me.json(); + return { + token: payload.token, + userId: identity.user.id, + organizationId: identity.orgs[0].id, + }; +} + +function seedDeadLetter({ organizationId, eventId, subscriptionId, customerId }) { + db.prepare(` + INSERT INTO billing_stripe_customers(customer_id, organization_id, first_observed_at_ms) + VALUES(?,?,?) + `).run(customerId, organizationId, 1_000); + db.prepare(` + INSERT INTO billing_stripe_subscriptions(subscription_id, customer_id, first_observed_at_ms) + VALUES(?,?,?) + `).run(subscriptionId, customerId, 1_000); + db.prepare(` + INSERT INTO billing_stripe_webhook_events( + event_id, provider_created_at_sec, event_type, object_id, object_type, + api_version, request_id, payload_sha256, first_received_at_ms + ) VALUES(?,?,?,?,?,?,?,?,?) + `).run( + eventId, + 1_787_000_000, + 'customer.subscription.updated', + subscriptionId, + 'subscription', + '2025-03-31.basil', + null, + 'b'.repeat(64), + 1_000, + ); + db.prepare(` + INSERT INTO billing_stripe_reconciliation_triggers( + event_id, subscription_id, queued_at_ms, processing_state + ) VALUES(?,?,?,'pending') + `).run(eventId, subscriptionId, 1_000); + db.prepare(` + INSERT INTO billing_stripe_reconciliation_jobs( + event_id, processing_state, attempt_count, next_attempt_at_ms, + lease_token_sha256, lease_expires_at_ms, completed_at_ms, + last_error_code, claim_decision_id + ) VALUES(?,'dead_letter',5,2000,NULL,NULL,2000,'stripe_reconciliation_failed',NULL) + `).run(eventId); + db.prepare(` + INSERT INTO billing_stripe_reconciliation_attempts( + event_id, attempt_number, lease_started_at_ms, lease_expires_at_ms, + finished_at_ms, outcome, error_code + ) VALUES(?,5,1900,2000,2000,'dead_letter','stripe_reconciliation_failed') + `).run(eventId); +} + +const owner = await signup('recovery-owner@scopeweave.test', 'Recovery Owner'); +const member = await signup('recovery-member@scopeweave.test', 'Recovery Member'); +db.prepare('INSERT INTO memberships(org_id,user_id,role) VALUES(?,?,?)') + .run(owner.organizationId, member.userId, 'member'); + +const eventId = 'evt_api_dead_letter'; +const subscriptionId = 'sub_api_dead_letter'; +const customerId = 'cus_api_dead_letter'; +seedDeadLetter({ organizationId: owner.organizationId, eventId, subscriptionId, customerId }); + +let response = await request(`/api/orgs/${owner.organizationId}/billing/reconciliation/dead-letters`); +assert.equal(response.status, 401, 'dead-letter inspection requires authentication'); + +response = await request(`/api/orgs/${owner.organizationId}/billing/reconciliation/dead-letters`, { + headers: { authorization: `Bearer ${member.token}` }, +}); +assert.equal(response.status, 403, 'ordinary members cannot inspect billing recovery operations'); + +const ownerAuth = { authorization: `Bearer ${owner.token}` }; +response = await request(`/api/orgs/${owner.organizationId}/billing/reconciliation/dead-letters`, { + headers: ownerAuth, +}); +assert.equal(response.status, 200, 'workspace owner can inspect its dead-letter backlog'); +let payload = await response.json(); +assert.deepEqual(payload, { + deadLetters: [{ + eventId, + subscriptionId, + attemptCount: 5, + completedAtMs: 2_000, + lastErrorCode: 'stripe_reconciliation_failed', + }], +}); + +response = await request( + `/api/orgs/${member.organizationId}/billing/reconciliation/dead-letters/${eventId}/retry`, + { + method: 'POST', + headers: { authorization: `Bearer ${member.token}` }, + body: jsonBody({ evidenceReference: 'INC-foreign-tenant' }), + }, +); +assert.equal(response.status, 404, 'a foreign workspace cannot learn or recover another tenant dead letter'); + +response = await request( + `/api/orgs/${owner.organizationId}/billing/reconciliation/dead-letters/${eventId}/retry`, + { + method: 'POST', + headers: ownerAuth, + body: jsonBody({ evidenceReference: 'bad\nreference' }), + }, +); +assert.equal(response.status, 400, 'recovery requires a bounded control-free evidence reference'); + +const retryUrl = `http://scopeweave.test/api/orgs/${owner.organizationId}/billing/reconciliation/dead-letters/${eventId}/retry`; +const oversizedRecoveryBody = jsonBody({ evidenceReference: 'x'.repeat(5_000) }); +response = await app.request(new Request(retryUrl, { + method: 'POST', + headers: { + ...ownerAuth, + 'content-type': 'application/json', + 'content-length': String(Buffer.byteLength(oversizedRecoveryBody)), + }, + body: oversizedRecoveryBody, +})); +assert.equal(response.status, 413, 'declared oversized recovery JSON fails before parsing'); +assert.deepEqual(await response.json(), { error: 'stripe_reconciliation_recovery_body_too_large' }); + +const encoder = new TextEncoder(); +const oversizedRecoveryStream = new ReadableStream({ + start(controller) { + controller.enqueue(encoder.encode('{"evidenceReference":"')); + controller.enqueue(encoder.encode('y'.repeat(5_000))); + controller.enqueue(encoder.encode('"}')); + controller.close(); + }, +}); +const streamedRequest = new Request(retryUrl, { + method: 'POST', + headers: { ...ownerAuth, 'content-type': 'application/json' }, + body: oversizedRecoveryStream, + duplex: 'half', +}); +assert.equal(streamedRequest.headers.get('content-length'), null, 'stream regression has no declared byte length'); +response = await app.request(streamedRequest); +assert.equal(response.status, 413, 'streamed oversized recovery JSON fails at the byte ceiling'); +assert.deepEqual(await response.json(), { error: 'stripe_reconciliation_recovery_body_too_large' }); + +const originalFetch = globalThis.fetch; +let providerCalls = 0; +const nowSec = Math.floor(Date.now() / 1000); +globalThis.fetch = async (url, options) => { + providerCalls += 1; + assert.equal(url, `https://api.stripe.com/v1/subscriptions/${subscriptionId}`); + assert.equal(options.method, 'GET'); + assert.equal(options.redirect, 'error'); + assert.equal(options.headers.authorization, 'Bearer sk_test_dead_letter_recovery'); + const body = JSON.stringify({ + id: subscriptionId, + object: 'subscription', + customer: customerId, + status: 'trialing', + metadata: { orgId: String(owner.organizationId) }, + cancel_at_period_end: false, + current_period_start: nowSec - 60, + current_period_end: nowSec + 3_600, + canceled_at: null, + ended_at: null, + trial_end: nowSec + 3_600, + latest_invoice: null, + items: { data: [{ price: { id: 'price_scopeweave_trial' } }] }, + }); + return new Response(body, { + status: 200, + headers: { + 'content-type': 'application/json; charset=utf-8', + 'content-length': String(Buffer.byteLength(body)), + }, + }); +}; + +try { + response = await request( + `/api/orgs/${owner.organizationId}/billing/reconciliation/dead-letters/${eventId}/retry`, + { + method: 'POST', + headers: ownerAuth, + body: jsonBody({ evidenceReference: 'INC-2026-API-0001' }), + }, + ); + assert.equal(response.status, 200, 'authorized recovery performs one current-provider reconciliation'); + payload = await response.json(); + assert.equal(payload.status, 'succeeded'); + assert.equal(payload.replayed, false); + assert.equal(payload.eventId, eventId); + assert.equal(payload.subscriptionId, subscriptionId); + assert.equal(payload.attemptNumber, 6); + assert.ok(Number.isSafeInteger(payload.recoveryId) && payload.recoveryId > 0); + assert.ok(Number.isSafeInteger(payload.claimDecisionId) && payload.claimDecisionId > 0); + assert.equal(providerCalls, 1); + + response = await request( + `/api/orgs/${owner.organizationId}/billing/reconciliation/dead-letters/${eventId}/retry`, + { + method: 'POST', + headers: ownerAuth, + body: jsonBody({ evidenceReference: 'INC-2026-API-0001' }), + }, + ); + assert.equal(response.status, 200, 'replaying the same operator evidence is idempotent'); + const replay = await response.json(); + assert.equal(replay.replayed, true); + assert.equal(replay.recoveryId, payload.recoveryId); + assert.equal(replay.claimDecisionId, payload.claimDecisionId); + assert.equal(providerCalls, 1, 'idempotent replay does not repeat Stripe provider reads'); +} finally { + globalThis.fetch = originalFetch; +} + +response = await request(`/api/orgs/${owner.organizationId}/billing/reconciliation/dead-letters`, { + headers: ownerAuth, +}); +payload = await response.json(); +assert.deepEqual(payload.deadLetters, [], 'successful manual recovery clears the dead-letter backlog'); + +const durableRecovery = db.prepare(` + SELECT actor_user_id, evidence_reference, attempt_number + FROM billing_stripe_reconciliation_recoveries + WHERE event_id = ? +`).get(eventId); +assert.deepEqual({ ...durableRecovery }, { + actor_user_id: owner.userId, + evidence_reference: 'INC-2026-API-0001', + attempt_number: 6, +}); + +db.close(); diff --git a/tests/fuzz/stripeReconciliationRecovery.fuzz.mjs b/tests/fuzz/stripeReconciliationRecovery.fuzz.mjs new file mode 100644 index 00000000..6b6b3d5e --- /dev/null +++ b/tests/fuzz/stripeReconciliationRecovery.fuzz.mjs @@ -0,0 +1,111 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; + +import { + StripeReconciliationRecoveryError, + retryStripeReconciliationDeadLetter, +} from '../../server/stripe_reconciliation_recovery.mjs'; + +function claimedRecovery() { + return { + status: 'processing', + replayed: false, + recoveryId: 19, + eventId: 'evt_completion_uncertain', + subscriptionId: 'sub_completion_uncertain', + organizationId: 7, + attemptNumber: 6, + leaseToken: 'manual_recovery_token_00000019', + leaseExpiresAtMs: 90_000, + }; +} + +test('provider success followed by uncertain completion never starts a contradictory failure transition', async () => { + let failureTransitions = 0; + const recoveryRepository = { + claimDeadLetterRecovery() { + return claimedRecovery(); + }, + completeRecovery() { + throw new StripeReconciliationRecoveryError( + 'stripe_reconciliation_recovery_state_uncertain', + 500, + ); + }, + failRecovery() { + failureTransitions += 1; + return { status: 'dead_letter' }; + }, + }; + const workerRepository = { + complete() {}, + fail() {}, + }; + + await assert.rejects( + retryStripeReconciliationDeadLetter({ + recoveryRepository, + workerRepository, + reconcile: async () => ({ + organizationId: 7, + subscriptionId: 'sub_completion_uncertain', + claimDecisionId: 73, + }), + organizationId: 7, + eventId: 'evt_completion_uncertain', + actorUserId: 11, + evidenceReference: 'INC-uncertain-completion', + }), + (error) => error instanceof StripeReconciliationRecoveryError + && error.code === 'stripe_reconciliation_recovery_state_uncertain' + && error.status === 500, + ); + assert.equal(failureTransitions, 0); +}); + +test('provider reconciliation failure still performs one bounded failure transition', async () => { + let failureTransitions = 0; + const claim = claimedRecovery(); + const recoveryRepository = { + claimDeadLetterRecovery() { + return claim; + }, + completeRecovery() { + throw new Error('completion must not run after provider failure'); + }, + failRecovery({ errorCode }) { + failureTransitions += 1; + assert.equal(errorCode, 'stripe_provider_temporarily_unavailable'); + return { status: 'dead_letter' }; + }, + }; + const workerRepository = { + complete() {}, + fail() {}, + }; + const providerError = new Error('provider body must not escape'); + providerError.code = 'stripe_provider_temporarily_unavailable'; + + const result = await retryStripeReconciliationDeadLetter({ + recoveryRepository, + workerRepository, + reconcile: async () => { + throw providerError; + }, + organizationId: 7, + eventId: claim.eventId, + actorUserId: 11, + evidenceReference: 'INC-provider-failure', + }); + + assert.deepEqual(result, { + status: 'dead_letter', + replayed: false, + recoveryId: claim.recoveryId, + eventId: claim.eventId, + subscriptionId: claim.subscriptionId, + attemptNumber: claim.attemptNumber, + errorCode: 'stripe_provider_temporarily_unavailable', + }); + assert.equal(failureTransitions, 1); +}); diff --git a/tests/unit/stripe-reconciliation-dead-letter-recovery.test.mjs b/tests/unit/stripe-reconciliation-dead-letter-recovery.test.mjs new file mode 100644 index 00000000..c45a77cc --- /dev/null +++ b/tests/unit/stripe-reconciliation-dead-letter-recovery.test.mjs @@ -0,0 +1,483 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { DatabaseSync } from 'node:sqlite'; + +import { installStripeSubscriptionObservationSchema } from '../../server/stripe_subscription_observation_ledger.mjs'; +import { installStripeWebhookEventSchema } from '../../server/stripe_webhook_event_ledger.mjs'; +import { installStripeWebhookReconciliationQueueSchema } from '../../server/stripe_webhook_reconciliation_queue.mjs'; +import { + createSqliteStripeReconciliationWorkerRepository, + installStripeReconciliationWorkerSchema, +} from '../../server/stripe_reconciliation_worker.mjs'; +import { + StripeReconciliationRecoveryError, + createSqliteStripeReconciliationRecoveryRepository, + installStripeReconciliationRecoverySchema, + retryStripeReconciliationDeadLetter, +} from '../../server/stripe_reconciliation_recovery.mjs'; + +function createDatabase() { + const database = new DatabaseSync(':memory:'); + database.exec('PRAGMA foreign_keys = ON'); + database.exec(` + CREATE TABLE users ( + id INTEGER PRIMARY KEY, + email TEXT UNIQUE NOT NULL + ); + CREATE TABLE orgs ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL, + plan TEXT NOT NULL DEFAULT 'free' + ); + `); + installStripeWebhookEventSchema(database); + installStripeSubscriptionObservationSchema(database); + installStripeWebhookReconciliationQueueSchema(database); + installStripeReconciliationWorkerSchema(database); + installStripeReconciliationRecoverySchema(database); + return database; +} + +function seedDeadLetter(database, { + eventId = 'evt_dead_letter', + subscriptionId = 'sub_dead_letter', + customerId = 'cus_dead_letter', + organizationId = 7, + actorUserId = 11, + attemptCount = 5, +} = {}) { + database.prepare('INSERT OR IGNORE INTO users(id,email) VALUES(?,?)') + .run(actorUserId, `operator-${actorUserId}@scopeweave.test`); + database.prepare('INSERT OR IGNORE INTO orgs(id,name,plan) VALUES(?,?,?)') + .run(organizationId, `Org ${organizationId}`, 'free'); + database.prepare(` + INSERT OR IGNORE INTO billing_stripe_customers(customer_id, organization_id, first_observed_at_ms) + VALUES(?,?,?) + `).run(customerId, organizationId, 1_000); + database.prepare(` + INSERT OR IGNORE INTO billing_stripe_subscriptions(subscription_id, customer_id, first_observed_at_ms) + VALUES(?,?,?) + `).run(subscriptionId, customerId, 1_000); + database.prepare(` + INSERT INTO billing_stripe_webhook_events( + event_id, provider_created_at_sec, event_type, object_id, object_type, + api_version, request_id, payload_sha256, first_received_at_ms + ) VALUES(?,?,?,?,?,?,?,?,?) + `).run( + eventId, + 1_787_000_000, + 'customer.subscription.updated', + subscriptionId, + 'subscription', + '2025-03-31.basil', + null, + 'a'.repeat(64), + 1_000, + ); + database.prepare(` + INSERT INTO billing_stripe_reconciliation_triggers( + event_id, subscription_id, queued_at_ms, processing_state + ) VALUES(?,?,?,'pending') + `).run(eventId, subscriptionId, 1_000); + database.prepare(` + INSERT INTO billing_stripe_reconciliation_jobs( + event_id, processing_state, attempt_count, next_attempt_at_ms, + lease_token_sha256, lease_expires_at_ms, completed_at_ms, + last_error_code, claim_decision_id + ) VALUES(?,'dead_letter',?,?,NULL,NULL,?,?,NULL) + `).run(eventId, attemptCount, 2_000, 2_000, 'stripe_reconciliation_failed'); + database.prepare(` + INSERT INTO billing_stripe_reconciliation_attempts( + event_id, attempt_number, lease_started_at_ms, lease_expires_at_ms, + finished_at_ms, outcome, error_code + ) VALUES(?,?,?,?,?,'dead_letter','stripe_reconciliation_failed') + `).run(eventId, attemptCount, 1_900, 2_000, 2_000); +} + +function tokenSequence() { + let index = 0; + return () => `manual_recovery_token_${String(++index).padStart(8, '0')}`; +} + +test('operator recovery lists tenant dead letters, retries one exact event, and is idempotent by evidence reference', async () => { + const database = createDatabase(); + seedDeadLetter(database); + let nowMs = 3_000; + const recoveryRepository = createSqliteStripeReconciliationRecoveryRepository(database, { + now: () => nowMs, + randomToken: tokenSequence(), + leaseMs: 30_000, + }); + const workerRepository = createSqliteStripeReconciliationWorkerRepository(database, { + now: () => nowMs, + }); + + assert.deepEqual(recoveryRepository.listDeadLetters({ organizationId: 7 }), [{ + eventId: 'evt_dead_letter', + subscriptionId: 'sub_dead_letter', + attemptCount: 5, + completedAtMs: 2_000, + lastErrorCode: 'stripe_reconciliation_failed', + }]); + assert.deepEqual(recoveryRepository.listDeadLetters({ organizationId: 8 }), []); + + let reconcileCalls = 0; + const first = await retryStripeReconciliationDeadLetter({ + recoveryRepository, + workerRepository, + reconcile: async (input) => { + reconcileCalls += 1; + assert.deepEqual(input, { + organizationId: 7, + subscriptionId: 'sub_dead_letter', + sourceEventId: 'evt_dead_letter', + secretKey: 'server-owned-secret', + }); + return { + organizationId: 7, + subscriptionId: 'sub_dead_letter', + claimDecisionId: 41, + }; + }, + reconciliationDependencies: { secretKey: 'server-owned-secret' }, + organizationId: 7, + eventId: 'evt_dead_letter', + actorUserId: 11, + evidenceReference: 'INC-2026-0042', + }); + assert.deepEqual(first, { + status: 'succeeded', + replayed: false, + recoveryId: 1, + eventId: 'evt_dead_letter', + subscriptionId: 'sub_dead_letter', + attemptNumber: 6, + claimDecisionId: 41, + }); + assert.equal(reconcileCalls, 1); + assert.deepEqual(recoveryRepository.listDeadLetters({ organizationId: 7 }), []); + + const job = database.prepare(` + SELECT processing_state, attempt_count, claim_decision_id, completed_at_ms, last_error_code + FROM billing_stripe_reconciliation_jobs WHERE event_id = ? + `).get('evt_dead_letter'); + assert.deepEqual({ ...job }, { + processing_state: 'succeeded', + attempt_count: 6, + claim_decision_id: 41, + completed_at_ms: nowMs, + last_error_code: null, + }); + const attempts = database.prepare(` + SELECT attempt_number, outcome, error_code + FROM billing_stripe_reconciliation_attempts + WHERE event_id = ? ORDER BY attempt_number + `).all('evt_dead_letter').map((row) => ({ ...row })); + assert.deepEqual(attempts, [ + { attempt_number: 5, outcome: 'dead_letter', error_code: 'stripe_reconciliation_failed' }, + { attempt_number: 6, outcome: 'succeeded', error_code: null }, + ]); + const recovery = database.prepare(` + SELECT event_id, attempt_number, actor_user_id, evidence_reference, requested_at_ms + FROM billing_stripe_reconciliation_recoveries WHERE recovery_id = 1 + `).get(); + assert.deepEqual({ ...recovery }, { + event_id: 'evt_dead_letter', + attempt_number: 6, + actor_user_id: 11, + evidence_reference: 'INC-2026-0042', + requested_at_ms: nowMs, + }); + + const replay = await retryStripeReconciliationDeadLetter({ + recoveryRepository, + workerRepository, + reconcile: async () => { + reconcileCalls += 1; + throw new Error('idempotent replay must not call provider reconciliation'); + }, + organizationId: 7, + eventId: 'evt_dead_letter', + actorUserId: 11, + evidenceReference: 'INC-2026-0042', + }); + assert.deepEqual(replay, { ...first, replayed: true }); + assert.equal(reconcileCalls, 1); + + database.close(); +}); + +test('failed manual recovery returns to dead-letter, suppresses unsafe exception text, and permits a new explicit recovery authority', async () => { + const database = createDatabase(); + seedDeadLetter(database, { eventId: 'evt_manual_retry', subscriptionId: 'sub_manual_retry' }); + let nowMs = 5_000; + const recoveryRepository = createSqliteStripeReconciliationRecoveryRepository(database, { + now: () => nowMs, + randomToken: tokenSequence(), + leaseMs: 30_000, + }); + const workerRepository = createSqliteStripeReconciliationWorkerRepository(database, { + now: () => nowMs, + }); + let calls = 0; + + const failure = new Error('provider exposed sk_live_must_not_persist'); + failure.code = 'stripe_provider_temporarily_unavailable'; + const failed = await retryStripeReconciliationDeadLetter({ + recoveryRepository, + workerRepository, + reconcile: async () => { + calls += 1; + throw failure; + }, + organizationId: 7, + eventId: 'evt_manual_retry', + actorUserId: 11, + evidenceReference: 'INC-2026-0043-attempt-1', + }); + assert.deepEqual(failed, { + status: 'dead_letter', + replayed: false, + recoveryId: 1, + eventId: 'evt_manual_retry', + subscriptionId: 'sub_manual_retry', + attemptNumber: 6, + errorCode: 'stripe_provider_temporarily_unavailable', + }); + assert.equal(calls, 1); + assert.equal(JSON.stringify(database.prepare(` + SELECT * FROM billing_stripe_reconciliation_attempts WHERE event_id = ? + `).all('evt_manual_retry')).includes('sk_live'), false); + + const replay = await retryStripeReconciliationDeadLetter({ + recoveryRepository, + workerRepository, + reconcile: async () => { + calls += 1; + throw new Error('must not run for the same evidence reference'); + }, + organizationId: 7, + eventId: 'evt_manual_retry', + actorUserId: 11, + evidenceReference: 'INC-2026-0043-attempt-1', + }); + assert.deepEqual(replay, { ...failed, replayed: true }); + assert.equal(calls, 1); + + nowMs = 6_000; + const succeeded = await retryStripeReconciliationDeadLetter({ + recoveryRepository, + workerRepository, + reconcile: async () => ({ + organizationId: 7, + subscriptionId: 'sub_manual_retry', + claimDecisionId: 73, + }), + organizationId: 7, + eventId: 'evt_manual_retry', + actorUserId: 11, + evidenceReference: 'INC-2026-0043-attempt-2', + }); + assert.equal(succeeded.status, 'succeeded'); + assert.equal(succeeded.attemptNumber, 7); + assert.equal(succeeded.claimDecisionId, 73); + + database.close(); +}); + +test('tenant isolation, bounded evidence references, in-progress replay, and transactional audit rollback fail closed', () => { + const database = createDatabase(); + seedDeadLetter(database, { eventId: 'evt_recovery_guard', subscriptionId: 'sub_recovery_guard' }); + database.prepare('INSERT INTO orgs(id,name,plan) VALUES(?,?,?)').run(8, 'Foreign Org', 'free'); + database.prepare('INSERT INTO users(id,email) VALUES(?,?)').run(12, 'foreign-operator@scopeweave.test'); + let nowMs = 7_000; + const recoveryRepository = createSqliteStripeReconciliationRecoveryRepository(database, { + now: () => nowMs, + randomToken: tokenSequence(), + leaseMs: 30_000, + }); + const workerRepository = createSqliteStripeReconciliationWorkerRepository(database, { + now: () => nowMs, + }); + + assert.throws( + () => recoveryRepository.claimDeadLetterRecovery({ + organizationId: 8, + eventId: 'evt_recovery_guard', + actorUserId: 12, + evidenceReference: 'INC-foreign', + }), + (error) => error instanceof StripeReconciliationRecoveryError + && error.code === 'stripe_reconciliation_dead_letter_not_found' + && error.status === 404, + ); + for (const evidenceReference of ['', ' ', 'bad\nreference', 'x'.repeat(257), null]) { + assert.throws( + () => recoveryRepository.claimDeadLetterRecovery({ + organizationId: 7, + eventId: 'evt_recovery_guard', + actorUserId: 11, + evidenceReference, + }), + (error) => error instanceof StripeReconciliationRecoveryError + && error.code === 'stripe_reconciliation_recovery_invalid', + ); + } + assert.throws( + () => recoveryRepository.listDeadLetters({ organizationId: 7, limit: 101 }), + (error) => error instanceof StripeReconciliationRecoveryError + && error.code === 'stripe_reconciliation_recovery_invalid', + ); + + const claimed = recoveryRepository.claimDeadLetterRecovery({ + organizationId: 7, + eventId: 'evt_recovery_guard', + actorUserId: 11, + evidenceReference: 'INC-in-progress', + }); + assert.equal(claimed.status, 'processing'); + assert.equal(claimed.replayed, false); + const replay = recoveryRepository.claimDeadLetterRecovery({ + organizationId: 7, + eventId: 'evt_recovery_guard', + actorUserId: 11, + evidenceReference: 'INC-in-progress', + }); + assert.deepEqual(replay, { + status: 'processing', + replayed: true, + recoveryId: claimed.recoveryId, + eventId: claimed.eventId, + subscriptionId: claimed.subscriptionId, + attemptNumber: claimed.attemptNumber, + }); + workerRepository.fail({ + eventId: claimed.eventId, + leaseToken: claimed.leaseToken, + errorCode: 'stripe_reconciliation_operator_cancelled', + }); + + seedDeadLetter(database, { + eventId: 'evt_recovery_rollback', + subscriptionId: 'sub_recovery_rollback', + customerId: 'cus_recovery_rollback', + }); + database.exec(` + CREATE TRIGGER fail_recovery_audit + BEFORE INSERT ON billing_stripe_reconciliation_recoveries + BEGIN + SELECT RAISE(ABORT, 'injected recovery audit failure'); + END; + `); + assert.throws(() => recoveryRepository.claimDeadLetterRecovery({ + organizationId: 7, + eventId: 'evt_recovery_rollback', + actorUserId: 11, + evidenceReference: 'INC-rollback', + }), /injected recovery audit failure/); + const rolledBackJob = database.prepare(` + SELECT processing_state, attempt_count, completed_at_ms, last_error_code + FROM billing_stripe_reconciliation_jobs WHERE event_id = ? + `).get('evt_recovery_rollback'); + assert.deepEqual({ ...rolledBackJob }, { + processing_state: 'dead_letter', + attempt_count: 5, + completed_at_ms: 2_000, + last_error_code: 'stripe_reconciliation_failed', + }); + assert.equal(database.prepare(` + SELECT COUNT(*) AS count FROM billing_stripe_reconciliation_attempts + WHERE event_id = ? AND attempt_number = 6 + `).get('evt_recovery_rollback').count, 0); + + database.close(); +}); + +test('expired manual recovery lease self-reaps to dead-letter and permits explicit retry without a scheduler', () => { + const database = createDatabase(); + seedDeadLetter(database, { + eventId: 'evt_recovery_expired', + subscriptionId: 'sub_recovery_expired', + customerId: 'cus_recovery_expired', + }); + let nowMs = 9_000; + const recoveryRepository = createSqliteStripeReconciliationRecoveryRepository(database, { + now: () => nowMs, + randomToken: tokenSequence(), + leaseMs: 100, + }); + + const claimed = recoveryRepository.claimDeadLetterRecovery({ + organizationId: 7, + eventId: 'evt_recovery_expired', + actorUserId: 11, + evidenceReference: 'INC-expired-1', + }); + assert.equal(claimed.status, 'processing'); + assert.equal(claimed.attemptNumber, 6); + + nowMs = claimed.leaseExpiresAtMs; + const expiredReplay = recoveryRepository.claimDeadLetterRecovery({ + organizationId: 7, + eventId: 'evt_recovery_expired', + actorUserId: 11, + evidenceReference: 'INC-expired-1', + }); + assert.deepEqual(expiredReplay, { + status: 'dead_letter', + replayed: true, + recoveryId: claimed.recoveryId, + eventId: claimed.eventId, + subscriptionId: claimed.subscriptionId, + attemptNumber: claimed.attemptNumber, + errorCode: 'stripe_reconciliation_lease_expired', + }); + + const job = database.prepare(` + SELECT processing_state, attempt_count, lease_token_sha256, lease_expires_at_ms, + completed_at_ms, last_error_code + FROM billing_stripe_reconciliation_jobs WHERE event_id = ? + `).get('evt_recovery_expired'); + assert.deepEqual({ ...job }, { + processing_state: 'dead_letter', + attempt_count: 6, + lease_token_sha256: null, + lease_expires_at_ms: null, + completed_at_ms: nowMs, + last_error_code: 'stripe_reconciliation_lease_expired', + }); + const attempt = database.prepare(` + SELECT finished_at_ms, outcome, error_code + FROM billing_stripe_reconciliation_attempts + WHERE event_id = ? AND attempt_number = 6 + `).get('evt_recovery_expired'); + assert.deepEqual({ ...attempt }, { + finished_at_ms: nowMs, + outcome: 'dead_letter', + error_code: 'stripe_reconciliation_lease_expired', + }); + const recovery = database.prepare(` + SELECT completed_at_ms, outcome, error_code, claim_decision_id + FROM billing_stripe_reconciliation_recoveries + WHERE recovery_id = ? + `).get(claimed.recoveryId); + assert.deepEqual({ ...recovery }, { + completed_at_ms: nowMs, + outcome: 'dead_letter', + error_code: 'stripe_reconciliation_lease_expired', + claim_decision_id: null, + }); + + nowMs += 1; + const retried = recoveryRepository.claimDeadLetterRecovery({ + organizationId: 7, + eventId: 'evt_recovery_expired', + actorUserId: 11, + evidenceReference: 'INC-expired-2', + }); + assert.equal(retried.status, 'processing'); + assert.equal(retried.replayed, false); + assert.equal(retried.attemptNumber, 7); + + database.close(); +});