Skip to content

Use openai api instead of Ollama #32

Description

@geoffroy-noel-ddh

Currently a few operators are designed to call an Ollama api:

  • answer_frames_ollama
  • answer_transcription_ollama

Here we propose to send the prompt in OpenAI format instead so the operators remain compatible with Ollama but also support other engines or LLM platforms.

Advantages:

  1. broader compatibility with inference engines or llm platforms
  2. reduced complexity & dependencies: we can just make that call with standard Python libraries without a container
  3. opportunity to move that Openai calling logic into a common ancestor operator

It also presents some difficulties as we need to preserve the following features that where supported directly (or easily) by Ollama API:

  • sends a seed (for reproducible outputs)
  • enable/disable thinking mode
  • specify the context size in case models are loaded dynamically (e.g. Ollama)
  • download the model on demand

Additional sub-tasks:

  • manage errors from inference API
  • make sure json results are post-processed properly
  • update readme
  • add param for API token

Metadata

Metadata

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions