Skip to content

[bug] A restore with another key can leave the restored server converting the backup's old AI endpoints #4801

Open

Description

@Quick104

What happened

A separate restore of a backup taken with another SECRET_KEY, run with --accept-lost-credentials, can leave the restored server with no AI provider list and nothing that tells the restore to reset the AI providers. The restored server's first start then runs the legacy AI endpoint conversion on what the backup still holds. It creates a keyless provider at the backup's old ai.base_url address and binds the text job to it. With recommendations on, it also creates an Ollama provider at http://ollama:11434 and binds embeddings to it. Neither was configured on the backed-up server at the time of the backup.

This follows PR #4800, which clears the AI job bindings together with the provider list and leaves an empty, already-converted list. That PR covers every restore where a binding or an AI credential is left. This issue is the case where neither is.

Steps to reproduce

Not reproduced live. Traced in source, plus a run of the conversion on the settings this state leaves (see Technical notes).

  1. On a server first set up on a build from before the AI provider list, configure a custom AI endpoint (ai.base_url). Upgrade it: the conversion moves the endpoint into the list and keeps the ai.base_url row.
  2. Unselect the provider for every AI job, and turn recommendations on without choosing an embedding provider, or do either one alone.
  3. Take a backup.
  4. On a server with a different SECRET_KEY, run silo backup restore --from <backup> --into <database URL> --database-only --accept-lost-credentials.
  5. Start Silo on the restored database and open AI Services.

Expected behavior

The restored server starts with no AI providers and no AI job bound, as it does when the backup still holds a binding or an AI key.

Silo version / commit

main at dec9d15 (the same on the branch of #4800)

Deployment

Not deployment-specific.

Area

Backups and restore

Clients affected

  • Other/not client-specific

Relevant logs

No logs available.

Technical notes

  • clearLostSettings (internal/backup/restore.go) runs ClearUnreadableSettings for a backup with another key under --accept-lost-credentials. That deletes every encrypted setting, including ai.providers and the AI keys, before the outbound carry runs.
  • CarryOutboundSettings (internal/backup/settings_outbound.go) clears the AI bundle only when one of its secret or binding rows is still set. With none left, it skips the bundle. No list row remains, so the list's conversion version reads as 0.
  • At start, convertLegacyAIProviders (cmd/silo/main.go) runs config.PlanAIProviderConversion. With no list and no binding stored, it adopts the leftover ai.base_url as a keyless custom provider and binds text. When recommendations.enabled is true or recommendations.embedding_lock is set, it adopts the Ollama default and binds embeddings. A run of PlanAIProviderConversion on such settings printed exactly those writes.
  • Possible repair: in a restore that doesn't take the live server's AI settings, reset the AI bundle even when its rows were already deleted. For example, also trigger the clear when the restored ai.providers row is missing.

Source: found in review of #4800

AI harness

Claude Code in T3 Code

AI tool(s)

Claude Code; T3 Code delegated review tasks (Codex, Claude)

AI model(s)

claude-opus-5-5; gpt-6.2-sol (review)

AI involvement

Fully AI-generated. Traced in source by the agent; not reproduced live, and not verified by a person.

Independent or adversarial review

Raised by the plan round and code review of #4800 (gpt-6.2-sol, claude-opus-5-5, claude-sonnet-5-5); the agent checked the trace against the source.

Confirmations

  • I searched open and recently closed issues and pull requests, and this is not already reported
  • I reproduced this myself on a real deployment
  • Any logs provided are raw apart from marked redactions and are not AI-summarized; if none were available, I said so above

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: P2Limited scope, workaround exists, or polish

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions