Skip to content

One undecryptable secret blanks the entire environment page for all users (permanent skeleton loaders) #948

Description

@jeinfra

Summary

On our self-hosted instance (v2.67.5), a single secret row with a corrupted value ciphertext made the environment secrets page unusable for every member of the app. The page stays on skeleton loaders forever, and the only signal is an unhandled rejection in the browser console:

Uncaught (in promise) Something went wrong: Error: invalid input

What happened

  • A secret's value ciphertext was truncated to exactly 16384 characters at write time (the writing client on our side was never identified; all three versions of the row were truncated the same way, months before anyone noticed). The truncation left the base64 segment with length % 4 == 1, which sodium.from_base64 rejects with invalid input.
  • The environment page decrypts every secret, including each secret's history, inside a single Promise.all (the decryptSecrets effect in frontend/app/[team]/apps/[app]/environments/[environment]/[[...path]]/page.tsx). One rejected row means the loaded state is never set, so every user of that environment gets skeletons forever with no error UI.
  • The CLI handles the same row differently: it silently skips what it cannot decode. phase secrets list showed N-1 secrets with no warning, so the console looked broken while the CLI looked healthy. That inconsistency masked the bad row for months and made diagnosis much harder.

Suggested fixes

  1. Decrypt each row independently in the console and render what succeeds, with a "cannot decrypt" badge on the broken row, instead of blanking the whole environment for everyone.
  2. Validate ciphertext shape server-side at write time. createSecret / editSecret / bulkProcessSecrets could cheaply reject anything that does not parse as ph:v<version>:<64-hex-pubkey>:<valid base64> before storing it.
  3. Whatever the failure, surface it in the UI rather than leaving an unhandled rejection and permanent skeletons.
  4. A warning from the CLI when it skips undecodable rows would have surfaced this months earlier.

Repro

  1. Create a secret, then corrupt it (via SQL or the API): truncate the stored value so the base64 segment has length % 4 == 1.
  2. Open the environment page as any member: permanent skeletons plus the invalid input rejection.
  3. Run phase secrets list: the row silently disappears and everything else lists fine.

Recovery for us was soft-deleting the row via the deleteSecret mutation (the CLI cannot address a row it cannot decode) and re-creating the secret. Happy to provide more detail.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions