Normalize plugin secret/connection refs + add usages SDK surface (#521)

* Add secret/connection usages SDK surface + UI placeholder

Plugins gain optional `usagesForSecret` / `usagesForConnection` callbacks
that the executor fans out across via `executor.secrets.usages(id)` and
`executor.connections.usages(id)`. Secret/connection removal is now
RESTRICT — refuses with `SecretInUseError` / `ConnectionInUseError` when
any plugin reports the id as in use, surfacing the count to the caller.

API gains GET `/secrets/:id/usages` and `/connections/:id/usages`
endpoints. React atoms and a small "Used by …" footer on each row in
the Secrets and Connections tabs round out the surface.

No plugin implements `usagesForSecret`/`usagesForConnection` yet, so
the footer renders nothing and remove still succeeds for everything.
Plugin migration lands in follow-up commits.

* Normalize graphql plugin secret/connection refs

Move secret/connection references out of the JSON columns on
graphql_source (auth, headers, query_params) into:
- two new flat columns auth_kind + auth_connection_id (indexed)
- two child tables graphql_source_header / graphql_source_query_param,
  each with kind/text_value/secret_id/secret_prefix and an indexed
  secret_id

This lets usagesForSecret and usagesForConnection answer "where is X
referenced?" with a single indexed SELECT instead of scanning JSON, and
secrets.remove / connections.remove now correctly RESTRICT when a
graphql source still references the id.

Migration 0007 backfills existing JSON via json_each + json_extract,
then drops the old columns. Three tests cover the backfill against a
hand-seeded pre-migration DB; three plugin-level tests cover usage
fan-out and remove RESTRICT.

* Normalize openapi plugin secret/connection refs

Move secret/connection references out of the JSON columns on
openapi_source and openapi_source_binding into:
- openapi_source_binding gains kind/secret_id/connection_id/text_value
  columns (replacing the discriminated-union json value)
- openapi_source_query_param: child table for query_params, mirrors
  graphql_source_query_param shape
- openapi_source_spec_fetch_header / spec_fetch_query_param: child
  tables for invocation_config.specFetchCredentials.{headers,queryParams}

Drops openapi_source.query_params and openapi_source.invocation_config
JSON columns. headers and oauth2 stay JSON because they hold slot
names rather than direct refs — the slots resolve through bindings,
which ARE normalized.

Plugin gains usagesForSecret / usagesForConnection that fan out across
all four locations in one indexed SELECT each. secrets.remove now
correctly RESTRICTs when an openapi binding still uses the secret.

Migration 0008 backfills via json_extract / json_each, then drops the
old columns. Three migration tests cover the backfill from a
hand-seeded pre-migration DB; two plugin-level tests cover usage
fan-out and remove RESTRICT.

* Normalize mcp plugin secret/connection refs

Move secret/connection references out of the JSON `mcp_source.config`
column into:
- Flat columns on mcp_source: auth_kind, auth_header_name,
  auth_secret_id, auth_secret_prefix, auth_connection_id,
  auth_client_id_secret_id, auth_client_secret_secret_id (each indexed
  where it carries a ref). Replaces the McpConnectionAuth json blob.
- mcp_source_header / mcp_source_query_param: child tables for the
  remote source's SecretBackedMap entries (same shape as graphql /
  openapi child tables).

The remaining structural fields (transport, endpoint, command, args,
etc.) stay as JSON in `config` because they're plugin-private and vary
by transport. mcp_binding.binding stays JSON too — McpToolBinding
carries no refs and inputSchema/outputSchema are arbitrary user JSON.

Plugin gains usagesForSecret / usagesForConnection. Migration 0009
backfills via json_extract / json_each, then strips the extracted
fields with json_remove. Three migration tests cover backfill from a
hand-seeded pre-migration DB; two plugin-level tests cover usage
fan-out across header-auth and oauth2 sources.

* Normalize google-discovery plugin secret/connection refs

Move secret/connection references out of the JSON
google_discovery_source.config column into:
- Flat columns on google_discovery_source: auth_kind,
  auth_connection_id, auth_client_id_secret_id,
  auth_client_secret_secret_id, auth_scopes (each indexed where it
  carries a ref). Replaces the GoogleDiscoveryAuth json.
- google_discovery_source_credential_header /
  google_discovery_source_credential_query_param: child tables for the
  optional GoogleDiscoveryFetchCredentials SecretBackedMap entries.

The remaining structural fields (discoveryUrl, service, version,
rootUrl, servicePath) stay in `config` as JSON — they're plugin-private
and never carry refs. google_discovery_binding stays JSON too: the
GoogleDiscoveryMethodBinding has no refs, and method input/output
schemas are arbitrary user data.

Plugin gains usagesForSecret / usagesForConnection. Migration 0010
backfills via json_extract / json_each, then strips the extracted
fields with json_remove. Three migration tests cover backfill from a
hand-seeded pre-migration DB; one plugin-level test covers usage
fan-out across auth_*_secret_id and auth_connection_id.

* Consolidate plugin normalization into single migration

Merge graphql/openapi/mcp/google-discovery normalization into one
migration per app: 0007_normalize_plugin_secret_refs (local sqlite)
and 0008_normalize_plugin_secret_refs (cloud pg, mcp-only since cloud
doesn't ship google-discovery).

Each test now imports a shared PRE_0007_SQL helper that seeds all four
plugins' parent tables at the pre-migration shape — needed because the
single consolidated migration touches every plugin in one transaction.

* Address review: legacy openapi OAuth + strict usage gate + shadow-aware delete

Three issues from review:

1. migrateOpenApi was early-returning when invocation_config column
   was missing (post-0007). Rows with legacy OAuth in oauth2 column
   silently lost their migration to a Connection. Fixed:
   - migrate-connections now reads oauth2 column even without
     invocation_config, and adapts its UPDATE statement accordingly.
   - Both 0007 (sqlite) and 0008 (pg) migrations COALESCE
     invocation_config.oauth2 into the oauth2 column before dropping
     invocation_config, so legacy data isn't lost mid-migration.

2. secrets.remove / connections.remove gated on a lenient usage helper
   that swallowed plugin failures into empty arrays — a transient
   plugin error could silently allow a delete that creates dangling
   refs. Added secretsUsagesStrict / connectionsUsagesStrict that
   propagate plugin failures as StorageFailure and wired them into the
   delete RESTRICT path. Display path keeps the lenient variant.

3. The RESTRICT check was global by id, conflicting with the documented
   shadowing model: deleting a user-scope override should reveal the
   outer fallback, but the global check blocked any delete whose id is
   referenced anywhere. Now scoped: only enforce RESTRICT when the
   delete would leave no row with that id (i.e., the reference would
   actually dangle).

Author: RhysSullivan

apps/cloud/drizzle/0008_normalize_plugin_secret_refs.sql

@@ -57,6 +57,13 @@
       "when": 1777567556847,
       "tag": "0007_military_young_avengers",
       "breakpoints": true
+    },
+    {
+      "idx": 8,
+      "version": "7",
+      "when": 1778004191000,
+      "tag": "0008_normalize_plugin_secret_refs",
+      "breakpoints": true
     }
   ]
-}
+}

apps/cloud/drizzle/meta/0008_snapshot.json

@@ -122,9 +122,7 @@ export const openapi_source = pgTable("openapi_source", {
   source_url: text('source_url'),
   base_url: text('base_url'),
   headers: jsonb('headers'),
-  query_params: jsonb('query_params'),
-  oauth2: jsonb('oauth2'),
-  invocation_config: jsonb('invocation_config').notNull()
+  oauth2: jsonb('oauth2')
 }, (table) => [
   primaryKey({ columns: [table.scope_id, table.id] }),
   index("openapi_source_scope_id_idx").on(table.scope_id),
@@ -147,25 +145,121 @@ export const openapi_source_binding = pgTable("openapi_source_binding", {
   source_scope_id: text('source_scope_id').notNull(),
   target_scope_id: text('target_scope_id').notNull(),
   slot: text('slot').notNull(),
-  value: jsonb('value').notNull(),
+  kind: text('kind', { enum: ['secret', 'connection', 'text'] }).notNull(),
+  secret_id: text('secret_id'),
+  connection_id: text('connection_id'),
+  text_value: text('text_value'),
   created_at: timestamp('created_at').notNull(),
   updated_at: timestamp('updated_at').notNull()
 }, (table) => [
   index("openapi_source_binding_source_id_idx").on(table.source_id),
   index("openapi_source_binding_source_scope_id_idx").on(table.source_scope_id),
   index("openapi_source_binding_target_scope_id_idx").on(table.target_scope_id),
   index("openapi_source_binding_slot_idx").on(table.slot),
+  index("openapi_source_binding_secret_id_idx").on(table.secret_id),
+  index("openapi_source_binding_connection_id_idx").on(table.connection_id),
+]);
+
+export const openapi_source_query_param = pgTable("openapi_source_query_param", {
+  id: text('id').notNull(),
+  scope_id: text('scope_id').notNull(),
+  source_id: text('source_id').notNull(),
+  name: text('name').notNull(),
+  kind: text('kind', { enum: ['text', 'secret'] }).notNull(),
+  text_value: text('text_value'),
+  secret_id: text('secret_id'),
+  secret_prefix: text('secret_prefix')
+}, (table) => [
+  primaryKey({ columns: [table.scope_id, table.id] }),
+  index("openapi_source_query_param_scope_id_idx").on(table.scope_id),
+  index("openapi_source_query_param_source_id_idx").on(table.source_id),
+  index("openapi_source_query_param_secret_id_idx").on(table.secret_id),
+]);
+
+export const openapi_source_spec_fetch_header = pgTable("openapi_source_spec_fetch_header", {
+  id: text('id').notNull(),
+  scope_id: text('scope_id').notNull(),
+  source_id: text('source_id').notNull(),
+  name: text('name').notNull(),
+  kind: text('kind', { enum: ['text', 'secret'] }).notNull(),
+  text_value: text('text_value'),
+  secret_id: text('secret_id'),
+  secret_prefix: text('secret_prefix')
+}, (table) => [
+  primaryKey({ columns: [table.scope_id, table.id] }),
+  index("openapi_source_spec_fetch_header_scope_id_idx").on(table.scope_id),
+  index("openapi_source_spec_fetch_header_source_id_idx").on(table.source_id),
+  index("openapi_source_spec_fetch_header_secret_id_idx").on(table.secret_id),
+]);
+
+export const openapi_source_spec_fetch_query_param = pgTable("openapi_source_spec_fetch_query_param", {
+  id: text('id').notNull(),
+  scope_id: text('scope_id').notNull(),
+  source_id: text('source_id').notNull(),
+  name: text('name').notNull(),
+  kind: text('kind', { enum: ['text', 'secret'] }).notNull(),
+  text_value: text('text_value'),
+  secret_id: text('secret_id'),
+  secret_prefix: text('secret_prefix')
+}, (table) => [
+  primaryKey({ columns: [table.scope_id, table.id] }),
+  index("openapi_source_spec_fetch_query_param_scope_id_idx").on(table.scope_id),
+  index("openapi_source_spec_fetch_query_param_source_id_idx").on(table.source_id),
+  index("openapi_source_spec_fetch_query_param_secret_id_idx").on(table.secret_id),
 ]);
 
 export const mcp_source = pgTable("mcp_source", {
   id: text('id').notNull(),
   scope_id: text('scope_id').notNull(),
   name: text('name').notNull(),
   config: jsonb('config').notNull(),
+  auth_kind: text('auth_kind', { enum: ['none', 'header', 'oauth2'] }).default("none").notNull(),
+  auth_header_name: text('auth_header_name'),
+  auth_secret_id: text('auth_secret_id'),
+  auth_secret_prefix: text('auth_secret_prefix'),
+  auth_connection_id: text('auth_connection_id'),
+  auth_client_id_secret_id: text('auth_client_id_secret_id'),
+  auth_client_secret_secret_id: text('auth_client_secret_secret_id'),
   created_at: timestamp('created_at').notNull()
 }, (table) => [
   primaryKey({ columns: [table.scope_id, table.id] }),
   index("mcp_source_scope_id_idx").on(table.scope_id),
+  index("mcp_source_auth_secret_id_idx").on(table.auth_secret_id),
+  index("mcp_source_auth_connection_id_idx").on(table.auth_connection_id),
+  index("mcp_source_auth_client_id_secret_id_idx").on(table.auth_client_id_secret_id),
+  index("mcp_source_auth_client_secret_secret_id_idx").on(table.auth_client_secret_secret_id),
+]);
+
+export const mcp_source_header = pgTable("mcp_source_header", {
+  id: text('id').notNull(),
+  scope_id: text('scope_id').notNull(),
+  source_id: text('source_id').notNull(),
+  name: text('name').notNull(),
+  kind: text('kind', { enum: ['text', 'secret'] }).notNull(),
+  text_value: text('text_value'),
+  secret_id: text('secret_id'),
+  secret_prefix: text('secret_prefix')
+}, (table) => [
+  primaryKey({ columns: [table.scope_id, table.id] }),
+  index("mcp_source_header_scope_id_idx").on(table.scope_id),
+  index("mcp_source_header_source_id_idx").on(table.source_id),
+  index("mcp_source_header_secret_id_idx").on(table.secret_id),
+]);
+
+export const mcp_source_query_param = pgTable("mcp_source_query_param", {
+  id: text('id').notNull(),
+  scope_id: text('scope_id').notNull(),
+  source_id: text('source_id').notNull(),
+  name: text('name').notNull(),
+  kind: text('kind', { enum: ['text', 'secret'] }).notNull(),
+  text_value: text('text_value'),
+  secret_id: text('secret_id'),
+  secret_prefix: text('secret_prefix')
+}, (table) => [
+  primaryKey({ columns: [table.scope_id, table.id] }),
+  index("mcp_source_query_param_scope_id_idx").on(table.scope_id),
+  index("mcp_source_query_param_source_id_idx").on(table.source_id),
+  index("mcp_source_query_param_secret_id_idx").on(table.secret_id),
 ]);
 
 export const mcp_binding = pgTable("mcp_binding", {
@@ -185,12 +279,44 @@ export const graphql_source = pgTable("graphql_source", {
   scope_id: text('scope_id').notNull(),
   name: text('name').notNull(),
   endpoint: text('endpoint').notNull(),
-  headers: jsonb('headers'),
-  query_params: jsonb('query_params'),
-  auth: jsonb('auth')
+  auth_kind: text('auth_kind', { enum: ['none', 'oauth2'] }).default("none").notNull(),
+  auth_connection_id: text('auth_connection_id')
 }, (table) => [
   primaryKey({ columns: [table.scope_id, table.id] }),
   index("graphql_source_scope_id_idx").on(table.scope_id),
+  index("graphql_source_auth_connection_id_idx").on(table.auth_connection_id),
+]);
+
+export const graphql_source_header = pgTable("graphql_source_header", {
+  id: text('id').notNull(),
+  scope_id: text('scope_id').notNull(),
+  source_id: text('source_id').notNull(),
+  name: text('name').notNull(),
+  kind: text('kind', { enum: ['text', 'secret'] }).notNull(),
+  text_value: text('text_value'),
+  secret_id: text('secret_id'),
+  secret_prefix: text('secret_prefix')
+}, (table) => [
+  primaryKey({ columns: [table.scope_id, table.id] }),
+  index("graphql_source_header_scope_id_idx").on(table.scope_id),
+  index("graphql_source_header_source_id_idx").on(table.source_id),
+  index("graphql_source_header_secret_id_idx").on(table.secret_id),
+]);
+
+export const graphql_source_query_param = pgTable("graphql_source_query_param", {
+  id: text('id').notNull(),
+  scope_id: text('scope_id').notNull(),
+  source_id: text('source_id').notNull(),
+  name: text('name').notNull(),
+  kind: text('kind', { enum: ['text', 'secret'] }).notNull(),
+  text_value: text('text_value'),
+  secret_id: text('secret_id'),
+  secret_prefix: text('secret_prefix')
+}, (table) => [
+  primaryKey({ columns: [table.scope_id, table.id] }),
+  index("graphql_source_query_param_scope_id_idx").on(table.scope_id),
+  index("graphql_source_query_param_source_id_idx").on(table.source_id),
+  index("graphql_source_query_param_secret_id_idx").on(table.secret_id),
 ]);
 
 export const graphql_operation = pgTable("graphql_operation", {