Skip to content

Feature request: allow disabling chat completions persistence (zero data retention) #2

Description

@meffmadd

🚀 Describe the new functionality needed

Allow operators to disable Chat Completions persistence by setting the inference store reference to null in a run config:

storage:
  stores:
    inference: null

Semantics when disabled:

  • No InferenceStore is constructed, no inference_store table is created, and no background write workers are started — so no chat completion payload is ever persisted.
  • Streaming and non-streaming Chat Completions keep working.
  • The history endpoints (list, retrieve, messages) surface that persistence is not configured (HTTP 501) rather than returning empty data or a 404.

💡 Why is this needed? What if we don't build it?

We host OGX internally as a shared Chat Completions service. By default the inference store persists every request/response payload, so the database fills up and there's no way to disable it. On top of that, we need to guarantee zero data retention for our users — something always-on persistence makes impossible.

Other thoughts

  • Backward compatible by default: omitting the key keeps persistence enabled; only an explicit inference: null disables it. This follows the same optional-store pattern the Responses store already uses.
  • Other stores (responses, datasets, eval, files, prompts, vector_io) are unaffected.
  • Is null the right way to express "disable this store", or would a boolean (e.g. inference.persist: false) be preferable? Note the responses store is already default=None, so null matches existing config semantics.
  • Should disabling persistence also apply to the Responses API / conversations (openai_conversations) store?
  • Would there be interest in a TTL/retention policy as an alternative to all-or-nothing opt-out?
  • Anything else that still writes to inference_store that this opt-out doesn't cover (cf. Don't write to inference_store when calling Responses API ogx-ai/ogx#2846)?

Implementation exists on branch disable-inference-store (PR: #1).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions