Skip to content

fix(platform): erase the subject's cloud grants, sync configs and blobs - #3122

Merged
Israeltheminer merged 1 commit into
mainfrom
fix/erasure-cloud-sync-cascade
Sep 3, 2026
Merged

fix(platform): erase the subject's cloud grants, sync configs and blobs#3122
Israeltheminer merged 1 commit into
mainfrom
fix/erasure-cloud-sync-cascade

Conversation

@Israeltheminer

Copy link
Copy Markdown
Collaborator

A GDPR erasure now removes the subject's cloud-import OAuth grant, their Drive and OneDrive sync configs, their video-link jobs, and the blobs behind their bare uploads. All of those survived a request that reported done.

Closes #3120.

Why

The 0.4 cascade ran 19 eraseSubject* arms. The 0.5 cascade runs 10 passes. Six categories had no pass at all:

Table What survived erasure
app.user_cloud_authorizations sealed OAuth2 access + refresh token
app.cloud_import_oauth_states an in-flight authorization
app.onedrive_sync_configs a sync still running under that grant
app.google_drive_sync_configs the same, for Drive
app.video_link_jobs pasted URL, video title, transcript ref
object storage every bare upload's bytes

The token row is the sharpest. 0042_cloud_import.sql describes it as an intentional Documents-import grant, sealed with the same envelope connector credentials use. It outlives the subject's membership.

The blob is the same class of miss. purgeDocument deletes the corpus entry and then the blob; the uploads pass deleted only the row.

What changed

  • purgeBlobArtefacts in retention/service.ts — the corpus-then-blob pair lifted out of purgeDocument, which now calls it. One implementation, two callers.
  • uploads purges each file's artefacts before deleting its row.
  • Three new passes: videoLinks, cloudGrants, syncConfigs.

video_link_jobs gets its own pass rather than folding into uploads: the job can own a blob when the linked file_metadata row never landed, and a welcome-page paste carries no thread, so neither the uploads pass nor the thread cascade reaches it.

Sync configs go because they name the member whose grant the sync runs under. Documents those syncs imported stay; that is org content, reached by the documents pass when the subject created it.

Risk

Each new pass loops with a knowledge-database round trip per row, where uploads was a single statement. The documents pass directly above it already has that shape, and the cascade runs as a background job.

Tests

integration-check.ts seeds all six categories for the erased subject and counts them into leftovers, which the existing assertion requires to be 0.

Observed against a throwaway Postgres, driving processErasure directly:

Cascade Result
this branch 0 rows survive; receipt done; counts uploads:1 videoLinks:1 cloudGrants:2 syncConfigs:2
origin/main 5 rows survive; receipt still done

The blob purge itself is not asserted. Observing it needs MinIO, which would not pull in this environment. It rides purgeBlobArtefacts, the same path purgeDocument already uses under CI's S3 lanes.

Scope

Four 0.4 arms still have no 0.5 pass: eraseSubjectPolicyAcknowledgements, eraseSubjectTwoFactorAttempts, eraseSubjectAutomationRuns and eraseSubjectWfExecutions. Their 0.5 tables need checking before anyone claims cascade parity. #3120 records that.

Gate: typecheck, oxlint --type-aware, oxfmt --check and lint:sast green.

@Israeltheminer

Copy link
Copy Markdown
Collaborator Author

Extended, and rescoped around #3135 to stop us colliding.

Dropped from this PR: the uploads pass. #3135 rewrites the same pass and does the blob half better than I had it — strict s3DeleteObject so a failed delete fails the pass and lands the receipt partial, where I was using the retention sweeps' best-effort helper that swallows the failure. That argument is right: a receipt saying done is a claim about the bytes. I also reverted my purgeBlobArtefacts refactor of retention/service.ts, so that file is untouched here now.

Added, after an audit of all 19 of 0.4's eraseSubject* arms against the 10 passes on main. Five more categories survived a done receipt:

Table What survived
app.notifications the org-level bells ABOUT the subject — subject_user_id was added for this pass and never read
app.automation_runs runs they started; input, output, trace, effects hold every node's resolved values
app.approvals + app.tasks.reviewer_user_id reviewer identity, and a live routing pointer at an erased user
app.login_attempts + app.login_block_counters the lockout trail, keyed by email
app.two_factor_attempts + app.two_factor_grace 2FA backoff, keyed by user id

The last two are global rather than org-scoped, so they are refused while the subject is still an active member of another organization — those counters protect those organizations too, and wiping them on one org's request would hand a multi-org user a lockout bypass. 0.4 had that gate; it paged Better Auth at a 256-membership cap and failed closed on truncation. SQL answers exactly, so neither the cap nor the guard is ported.

Review decisions are de-identified rather than deleted, to the same erased-user sentinel 0.4 used. The decision is the audit record of a governance gate, so the row stays; reviewer_user_id is live routing, so it is cleared.

Video-link blobs now delete strictly, matching #3135's argument rather than the retention idiom.

Tests

integration-check.ts seeds all five and counts four of them into leftovers, which the existing assertion requires to be 0. The review decision gets its own assertion, because de-identified-and-kept is not the same claim as deleted.

Observed against a throwaway Postgres, driving processErasure directly:

Cascade Result
this branch 8/8 assertions pass; counts orgNotifications:1 automationRuns:2 reviewDecisions:1 authState:3
origin/main 7 of 8 red

The eighth is an invariant, not a regression probe: it asserts the approvals ROW is still present, which holds on both sides.

Scope

Four findings from the same audit are deliberately not here, because they are different concerns and two of them are #3135's:

  • The legal hold is checked once before the cascade and never again, so a hold placed mid-cascade is ignored for every remaining pass. 0.4 re-read holds inside all 19 arms and its comment names the reason as FRCP 37(e) spoliation. Separate PR.
  • The receipt cannot distinguish "found nothing" from "never looked", and the drawer's Full breakdown renders blank on 0.5 because counts is now Record<string, number> while the renderer skips non-objects. Separate PR.
  • app.conversations retention never sweeps, so an admin-set externalConversations window is inert. Separate PR.
  • app.two_factor_attempts also lost its 30-day age sweep, and login_block_counters quietly went from 30 to 90 days. Separate PR.

policyAcknowledgements and wfExecutions are the two 0.4 arms with no 0.5 table, so their absence is not a regression. The docs still promise both, which is #3135's cross-class note.

Gate: typecheck, oxlint --type-aware, oxfmt --check and lint:sast green.

0.4 ran 19 eraseSubject arms; the 0.5 cascade runs 10 passes. Seven
categories of subject data survived a request that reported done:

- app.notifications, the ORG-level bells about the subject — a different
  table from the per-user inbox the notifications pass clears.
  subject_user_id was added for this and never read, and the lockout alert
  stamps it on the row carrying the subject's email and IP.
- app.automation_runs they started, whose input, output, trace and effects
  hold every node's resolved values.
- app.approvals and app.tasks.reviewer_user_id. The decision is KEPT and
  de-identified — it is the audit record of a governance gate — while the
  reviewer pointer is live routing and is cleared.
- app.login_attempts and app.login_block_counters, keyed by email.
- app.two_factor_attempts and app.two_factor_grace, keyed by user id.
- app.user_cloud_authorizations and app.cloud_import_oauth_states, the
  subject's sealed OAuth grant for Documents import.
- app.onedrive_sync_configs and app.google_drive_sync_configs, which name
  the member whose grant the sync runs under.
- app.video_link_jobs, which can own a blob even when the linked
  file_metadata row never landed.

The auth tables are global rather than org-scoped, so they are refused
while the subject is still an active member elsewhere: those counters
protect those organizations too, and wiping them on one org's request
hands a multi-org user a lockout bypass. 0.4 paged Better Auth at a
256-membership cap and failed closed on truncation; SQL answers exactly.

Video-link blobs delete strictly, following the uploads pass #3140 landed
rather than the retention sweeps' best-effort idiom: a receipt that says
done is a claim about the bytes.

Refs #3120.
@Israeltheminer
Israeltheminer force-pushed the fix/erasure-cloud-sync-cascade branch from fe7faf3 to 9441882 Compare September 3, 2026 12:17
@Israeltheminer

Copy link
Copy Markdown
Collaborator Author

Re-applied onto current main rather than rebased, because two erasure PRs landed in the meantime and one of them changed the primitive this was built on.

What #3135 and #3140 already fixed, so it is no longer here: the uploads pass and its blob leak. #3140 rewrote it with a strict s3DeleteObject — a failed delete throws, the receipt lands partial, the rows stay for Retry — plus a refcounted release seam and PurgeIncompleteError. That is better than the best-effort helper I had, so my version of that pass and my retention/service.ts refactor are both gone. That file is untouched now.

What is still needed, verified pass-by-pass against main: all seven. None of orgNotifications, automationRuns, reviewDecisions, authState, cloudGrants, syncConfigs or videoLinks exists on main today.

The videoLinks blob purge now follows #3140's idiom in the same file rather than my own — lazy store resolution inside the s3 branch, a ref-dedupe set, and a throw that fails the pass instead of swallowing the failure.

Tests

Extended integration-check.ts's existing erasure check: seeds all seven, counts six into leftovers (which the existing assertion requires to be 0), and gives the review decision its own assertion, because de-identified-and-kept is a different claim from deleted.

Run against a real Postgres and MinIO: 369/375, with the erasure check passing —

receipt=done, thread=0, leftovers=0, reviewDeidentified=1 (want 1), scrubbed=1

The six failures are the warm-MinIO bucket collision, two yt-dlp-dependent probes and three agent-lane probes — the same set a baseline run of unmodified main produces on this box.

The test caught two of my own fixture bugs before it caught anything else, which is worth recording. First run: leftovers=6, reviewDeidentified=0. I had seeded the seven categories after the cascade had already run, so the rows were never there to erase — it read as a code defect. Second: with the seed moved ahead of the filing, it passes. The lesson is the same one as #3171's fixture: a failing assertion is not evidence of a failing fix.

Still not here

Four findings from the same audit, all separate concerns and tracked in #3142: the legal hold is checked once rather than per pass, the receipt cannot distinguish "found nothing" from "never looked", the externalConversations retention window is inert, and two_factor_attempts lost its age sweep. The first two are #3144 (stacked on this branch, so it needs a rebase after this lands); the second two are #3143.

@Israeltheminer
Israeltheminer merged commit cd55e52 into main Sep 3, 2026
48 checks passed
@Israeltheminer
Israeltheminer deleted the fix/erasure-cloud-sync-cascade branch September 3, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: GDPR erasure leaves the subject's cloud tokens, sync configs and upload blobs behind

1 participant