Harden PayGate operations and recovery

.env.example

@@ -36,3 +36,28 @@ WEBHOOK_SECRET=
 
 # Tests only; do not enable in production.
 PAYGATE_TEST_MODE=false
+
+# Evidence privacy retention (Go duration syntax; 24h = 1 day)
+PAYGATE_RETENTION_ENABLED=true
+SMS_RAW_RETENTION=2160h
+RECONCILIATION_RAW_RETENTION=8760h
+AUDIT_RETENTION=17520h
+
+# Scheduled PocketBase backups. Empty cron disables automatic backups.
+# The default runs daily at 03:00 in the server timezone and keeps 14 archives.
+PAYGATE_BACKUP_CRON="0 3 * * *"
+PAYGATE_BACKUP_MAX_KEEP=14
+
+# Optional S3-compatible off-server backup storage.
+PAYGATE_BACKUP_S3_ENABLED=false
+PAYGATE_BACKUP_S3_BUCKET=
+PAYGATE_BACKUP_S3_REGION=
+PAYGATE_BACKUP_S3_ENDPOINT=
+PAYGATE_BACKUP_S3_ACCESS_KEY=
+PAYGATE_BACKUP_S3_SECRET=
+PAYGATE_BACKUP_S3_FORCE_PATH_STYLE=false
+
+# Optional signed operator-alert webhook for connector, capacity, webhook,
+# reconciliation and backup failures. Both values are required together.
+OPERATOR_ALERT_WEBHOOK_URL=
+OPERATOR_ALERT_WEBHOOK_SECRET=

ARCHITECTURE.md

@@ -362,3 +362,21 @@ web/                      React/Vite source
 10. Domain writes are backend-owned.
 11. Google Messages is a replaceable evidence connector, not the payment model.
 12. `/app/pb_data` must be persistent in production.
+
+## 20. Operational hardening
+
+The hardening migration adds six operator-domain collections:
+
+- `review_cases` links unresolved SMS or reconciliation evidence to candidate payments and records audited resolution;
+- `audit_events` stores attributable operator actions separately from application logs;
+- `reconciliation_runs` and `reconciliation_entries` preserve statement import results without auto-mutating payments;
+- `alerts` deduplicates connector/capacity/webhook/reconciliation/backup failures and maintains a durable signed notification delivery state;
+- `refunds` records manual refund intent, bank reference and lifecycle while enforcing aggregate amount bounds.
+
+Automatic matching and manual matching share the monetary invariants: exact amount equality, globally unique RRN and evidence time not earlier than payment creation. Manual review selects the intended record but cannot override those checks.
+
+Bank evidence time is authoritative for on-time/late classification. Ingestion time remains operational metadata. An SMS delivered after expiry can still confirm `paid` when its provider timestamp proves that the bank transaction occurred within the original payment window.
+
+Statement imports are intentionally report-first. Exact RRN+amount rows reconcile; contradictions create review cases. XLSX files are ZIP-validated before parsing to limit path traversal and decompression expansion.
+
+Backup configuration uses PocketBase's backup filesystem and cron. Archive verification reads every ZIP member. Restore drills extract to a temporary directory and run SQLite integrity checks without replacing production data.

OPERATIONS.md

@@ -0,0 +1,186 @@
+# PayGate Operations Runbook
+
+This runbook covers evidence review, statement reconciliation, alerts, refunds, retention, backup verification and production acceptance. PayGate remains fail-closed: an operator may choose the payment associated with reviewed evidence, but cannot override exact amount equality, bank-reference uniqueness or stale-evidence guards.
+
+## 1. Daily operator checks
+
+1. Open the dashboard and confirm Google Messages is `connected` and the phone is responsive.
+2. Confirm **Open reviews** and **Open alerts** are zero, or inspect every item.
+3. Check fingerprint capacity. Investigate warning pools at 70% and critical pools at 95%.
+4. Confirm the latest backup exists. Archive verification runs daily; failures create a critical alert.
+5. Review exhausted outgoing payment/refund webhooks.
+
+An optional signed operator-alert webhook can push open/resolved operational alerts to an external monitoring or notification system. Configure `OPERATOR_ALERT_WEBHOOK_URL` and `OPERATOR_ALERT_WEBHOOK_SECRET` together. Notifications are durable, deduplicated and retried; repeated health checks do not send repeated notifications for the same open alert.
+
+## 2. Evidence review
+
+Review cases are created for:
+
+- bank-credit-like SMS text that cannot be parsed;
+- a credit amount with no usable RRN/UTR;
+- exact credits that match no eligible payment;
+- ambiguous amount reuse;
+- a bank reference already associated with a different amount;
+- statement reconciliation conflicts.
+
+### Manual match procedure
+
+1. Inspect the original SMS or statement row and its provider/bank timestamp.
+2. Confirm the exact payable amount shown by PayGate.
+3. Confirm or enter the bank RRN/UTR/reference.
+4. Select the intended payment ID.
+5. Add a meaningful resolution note explaining the independent evidence used.
+6. Submit **Manually match exact evidence**.
+
+The server still rejects the action if:
+
+- the evidence amount differs from the payment payable amount;
+- the RRN is already assigned to another payment;
+- the bank evidence predates the payment;
+- the payment is already resolved by different evidence.
+
+Every resolution writes an immutable `audit_events` entry with actor, action, entity and details. Original evidence records are not deleted or rewritten beyond adding the reviewed reference/match result.
+
+## 3. Bank statement reconciliation
+
+Supported inputs:
+
+- CSV and TSV;
+- XLSX first worksheet;
+- up to 10 MiB, 10,000 credit rows and 64 columns.
+
+The importer recognizes common date, credit/deposit, narration, type and RRN/UPI-reference headings. Debit rows are ignored. XLSX archives are checked for unsafe paths, excessive entries and decompression expansion before parsing.
+
+Classification:
+
+- `matched` — bank RRN and exact amount already match a PayGate payment;
+- `duplicate` — the same RRN appears more than once in the imported statement;
+- `conflict` — same RRN/different amount, one amount candidate without a linked RRN, or multiple plausible historical candidates;
+- `unmatched` — no PayGate payment matches the credit;
+- `invalid` — required row values could not be parsed.
+
+Imports are report-only. They never mark payments paid automatically. Conflicting and UPI-like unmatched rows create review cases. Importing identical file bytes twice is rejected by SHA-256 hash.
+
+## 4. Refund lifecycle
+
+PayGate records refund work but does not move money.
+
+1. Create a refund only for a `paid` or `late` payment.
+2. Enter the amount in paise and the reason.
+3. Move the record to `processing` when the bank transfer is initiated.
+4. Enter the actual bank refund reference and mark it `completed` only after verification.
+
+The sum of active/completed refund records cannot exceed the received payment amount. Completed and cancelled records are terminal. Request and status events are signed and delivered through the normal durable outgoing webhook outbox:
+
+- `refund.requested`
+- `refund.processing`
+- `refund.completed`
+- `refund.failed`
+- `refund.cancelled`
+
+Consumers must still verify the signature and deduplicate the stable event ID.
+
+## 5. Fingerprint capacity
+
+Each whole-rupee requested amount has 99 paise fingerprints (`.01` through `.99`). Pending and quarantined records block reuse.
+
+- warning: 70 blocked fingerprints;
+- critical: 95 blocked fingerprints;
+- exhausted: all 99 blocked.
+
+Do not shorten quarantine merely to increase throughput. First measure real SMS delay and catch-up behavior. Prevent clients from repeatedly creating abandoned sessions and use idempotency keys for create retries.
+
+## 6. Evidence retention
+
+Defaults:
+
+- raw SMS identity/body fields: 90 days;
+- raw statement narration/row content: 365 days;
+- operator audit events: 730 days.
+
+After the SMS window, PayGate removes raw body/payload, sender, payer VPA and payer name while preserving source, provider ID, bank timestamp, amount, RRN and processing result. Statement rows retain amount, RRN, timestamp and reconciliation status after raw narration is removed.
+
+Configure with:
+
+```text
+PAYGATE_RETENTION_ENABLED=true
+SMS_RAW_RETENTION=2160h
+RECONCILIATION_RAW_RETENTION=8760h
+AUDIT_RETENTION=17520h
+```
+
+## 7. Backups and restore drills
+
+Automatic backups use PocketBase's transactionally generated `pb_data` archive and can be stored locally or in S3-compatible offsite storage.
+
+```text
+PAYGATE_BACKUP_CRON="0 3 * * *"
+PAYGATE_BACKUP_MAX_KEEP=14
+```
+
+For offsite storage configure all `PAYGATE_BACKUP_S3_*` variables. Secrets must be injected by the deployment system and never committed.
+
+Protection levels:
+
+1. scheduled backup creation;
+2. daily ZIP verification that reads every entry and confirms a database file exists;
+3. monthly non-destructive restore drill into a temporary directory, followed by SQLite `PRAGMA integrity_check` on every restored database.
+
+Manual commands:
+
+```bash
+payment-api backup-create
+payment-api backup-verify
+payment-api backup-restore-drill
+```
+
+The same actions are available from Operator UI → Settings. A drill does not replace the production volume and does not modify live data.
+
+## 8. Google Messages incident response
+
+### `reauth_required`
+
+The phone pairing/encryption keys may still be valid. Use the operator UI's Google login refresh with a fresh same-account Google Messages Web `config` Copy-as-cURL request. Do not unpair unless same-account reauthentication fails.
+
+### Disconnected
+
+- wait through the five-minute alert grace period;
+- confirm the phone is online and Google Messages is open/responding;
+- use **Reconnect**;
+- inspect redacted container logs for repeated auth or relay errors.
+
+### Phone unresponsive
+
+A warning opens after 15 minutes. Confirm battery optimization, connectivity and Google Messages background access on the paired phone.
+
+Never paste cookies or session data into issue trackers, logs or chat transcripts. The connector session is a credential and must remain inside the protected persistent volume.
+
+## 9. Production soak test
+
+Before relying on PayGate for real orders, run controlled payments over several days covering:
+
+- GPay, PhonePe, Paytm and another available UPI app;
+- delayed SMS while the payment itself occurred before expiry;
+- phone offline and later catch-up;
+- container restart and task replacement;
+- duplicate provider events and duplicate RRN;
+- wrong amount, missing RRN and unrecognized wording;
+- payment after expiry;
+- two concurrent sessions for the same whole-rupee price;
+- outgoing webhook failure/retry;
+- statement reconciliation and manual review;
+- partial refund and completed refund reference.
+
+Record payer action time, bank transaction time, SMS arrival time, ingestion time and final verification time. Do not reduce quarantine until the worst observed delay is understood.
+
+## 10. Release acceptance
+
+A release is production-ready only when:
+
+- frontend typecheck/build pass;
+- Go formatting, unit/integration tests, race tests and vet pass;
+- migration up/down is tested on a fresh database;
+- a production image builds and starts on a new temporary volume;
+- liveness/readiness, auth rejection, payment create, SMS match, review, reconciliation, refund, backup and restore-drill flows pass;
+- the temporary container is recreated on the same volume and state persists;
+- the current production volume is backed up and verified before schema cutover.

PLAN.md

@@ -62,7 +62,7 @@ This document records what was implemented and what must be proven before/after
 - [x] Accept/validate upstream-required cookie JSON, raw Cookie headers and DevTools Copy-as-cURL input without logging values.
 - [x] Keep QR pairing as a fallback and auto-refresh short-lived QR data.
 - [x] Refuse accidental re-pair over an existing session or another pairing in progress.
-- [ ] Complete a real phone Google-account/emoji pairing and verify reconnect/session persistence.
+- [x] Complete real-phone Google-account/emoji pairing, reconnect/session persistence and post-Tachyon-refresh connectivity verification.
 
 ### API/security
 
@@ -105,6 +105,21 @@ This document records what was implemented and what must be proven before/after
 - [x] Periodic auth refresh and 401 sign-out.
 - [x] UI create retries preserve idempotency key.
 
+
+### Operational hardening
+
+- [x] Persistent exception/review cases for parse, RRN, unmatched, ambiguity and reconciliation failures.
+- [x] Audited manual resolution that preserves exact amount, unique RRN and stale-evidence invariants.
+- [x] Bank occurrence-time classification so delayed SMS does not mislabel an on-time payment as late.
+- [x] CSV/TSV/XLSX statement reconciliation with duplicate-file/row detection and archive expansion limits.
+- [x] Fingerprint-pool capacity telemetry and warning/critical alerts.
+- [x] Persistent connector, webhook exhaustion, reconciliation and backup alerts.
+- [x] Optional signed durable operator-alert webhook with retries and anti-spam dedupe.
+- [x] Refund lifecycle records, aggregate amount bounds, audit trail and signed events without automatic fund movement.
+- [x] Configurable evidence retention/redaction.
+- [x] Scheduled local/S3-compatible backups, archive verification and temporary SQLite restore drills.
+- [x] Operator UI pages for reviews, reconciliation, alerts, refunds, audit, capacity and disaster recovery.
+
 ## Automated verification
 
 The final branch must pass all of these after the last code change:

README.md

@@ -139,6 +139,25 @@ Content-Type: application/json
 
 This compatibility route exists for migration only. The production default is disabled. Rotate the old relay to `SMS_WEBHOOK_SECRET` and `/api/events/sms`, then disable the legacy route.
 
+### Operational APIs
+
+Dashboard-authenticated operations:
+
+- `GET /api/capacity` — active/quarantined suffix-pool utilization;
+- `POST /api/review-cases/{id}/resolve` — audited review resolution/manual match;
+- `POST /api/reconciliation/import` — multipart CSV/TSV/XLSX statement import;
+- `GET /api/paygate/backups/status` — redacted archive status;
+- `POST /api/paygate/backups` — create a backup now;
+- `POST /api/paygate/backups/verify` — verify the latest archive;
+- `POST /api/paygate/backups/restore-drill` — temporary extraction plus SQLite integrity checks.
+
+API-key or dashboard-authenticated refund operations:
+
+- `POST /api/refunds` with an `Idempotency-Key`;
+- `POST /api/refunds/{id}/status`.
+
+Refund endpoints record operator/bank evidence only; they never initiate a bank transfer.
+
 ### Health
 
 - `GET /api/health` — PocketBase liveness endpoint, used by the container healthcheck.
@@ -155,9 +174,10 @@ Bank SMS processing follows these rules:
 3. reject automatic matching if amount or RRN is missing;
 4. treat an already-seen RRN with the same amount as an idempotent duplicate;
 5. treat the same RRN with a different amount as `RRN_AMOUNT_MISMATCH`;
-6. match a pending payment only by the exact payable paise amount and eligible timestamp;
-7. if no pending payment matches, check an expired/cancelled payment still in quarantine and mark it `late`;
-8. never silently assign ambiguous evidence.
+6. use the bank/provider occurrence timestamp to decide whether the transaction was on time, so delayed SMS delivery alone does not turn an on-time payment into `late`;
+7. match only the exact payable paise amount and eligible evidence window;
+8. if no on-time payment matches, check an expired/cancelled payment still in quarantine and mark a genuinely late transaction `late`;
+9. persist a review case instead of silently assigning incomplete, unmatched, contradictory or ambiguous evidence.
 
 Google Messages catch-up messages retain their provider message timestamp. Legacy relays that omit a timestamp are treated as arriving at ingestion time, so upgrading the legacy relay to send timestamps is recommended.
 
@@ -171,6 +191,11 @@ Events currently include:
 - `payment.late`
 - `payment.expired`
 - `payment.cancelled`
+- `refund.requested`
+- `refund.processing`
+- `refund.completed`
+- `refund.failed`
+- `refund.cancelled`
 
 Delivery records are written transactionally to `webhook_deliveries`; network I/O happens only after the transaction commits. The worker uses durable retries and recovers stale `sending` leases after process restarts.
 
@@ -199,6 +224,12 @@ The React UI at `/` provides:
 - realtime payment list/details;
 - cancellation;
 - SMS evidence records;
+- persistent evidence review and audited manual matching;
+- CSV/TSV/XLSX bank-statement reconciliation;
+- fingerprint-pool capacity monitoring;
+- operational alerts and signed notification delivery state;
+- manual refund lifecycle records;
+- backup creation, verification and temporary restore drills;
 - outgoing webhook delivery records;
 - Google Messages connector status, Google-account/emoji pairing and QR fallback controls;
 - safe non-secret configuration status.
@@ -234,7 +265,7 @@ Google-account pairing is the primary path. Browser cookie input is never logged
 
 Starting a new pairing is refused while another pairing is active or a valid session is already paired; explicitly cancel/unpair first.
 
-**Live phone Google-account/emoji pairing is the remaining connector acceptance test.** The connector code is integrated and unit-tested, but the private Google Messages protocol can change and must be validated with the actual phone before relying on it as the only ingestion source.
+The production connector has completed real-phone Google-account/emoji pairing, persisted its session across restarts and remained connected through the expected Tachyon authentication refresh boundary. This is strong acceptance evidence, but the private Google Messages protocol can still change; operational alerts and a reconciliation safety net remain necessary.
 
 ## Configuration
 
@@ -259,6 +290,14 @@ GMESSAGES_ENABLED=false
 GMESSAGES_SESSION_PATH=
 OUTGOING_WEBHOOK_URL=
 OUTGOING_WEBHOOK_SECRET=
+OPERATOR_ALERT_WEBHOOK_URL=
+OPERATOR_ALERT_WEBHOOK_SECRET=
+PAYGATE_RETENTION_ENABLED=true
+SMS_RAW_RETENTION=2160h
+RECONCILIATION_RAW_RETENTION=8760h
+AUDIT_RETENTION=17520h
+PAYGATE_BACKUP_CRON="0 3 * * *"
+PAYGATE_BACKUP_MAX_KEEP=14
 PB_DATA_DIR=./pb_data
 ```
 
@@ -326,5 +365,6 @@ A future proprietary/commercial distribution needs a separate licensing review r
 
 - `ARCHITECTURE.md` — implemented system design and invariants
 - `PLAN.md` — implementation/acceptance status
+- `OPERATIONS.md` — evidence review, reconciliation, refunds, alerts, backups and incident runbook
 - `RESEARCH.md` — technical research and constraints behind the design
 - `IMPLEMENTATION_SPEC.md` — rebuild requirements used during implementation