OpenLocalAI

A free, open-source, self-hosted, OpenAI-compatible AI engine — LLMs, voice, images, embeddings. No GPU required. No cloud. No telemetry.

Point any OpenAI SDK, app, or tool at OpenLocalAI. It speaks the same REST API. Prompts never leave your machine.

What it is
What works today
Requirements
Quickstart
Web UI
CLI
Why chat replies look like “echo”
Default model
Install a real LLM
Model gallery
Bring your own model
API reference
SDK examples
Backends
Configuration
Authentication
GPU
Docker
Kubernetes
Reverse proxy
Project layout
Development
Roadmap
License
Author & Maintainer

What it is

OpenLocalAI is a single Go binary that:

Serves an OpenAI-compatible HTTP API on http://localhost:8080/v1
Serves a Web UI on http://localhost:8080 (chat, models, playground)
Routes each model to a backend (echo, llamacpp, whispercpp, stablediffusion, piper)
Can download curated models from a local gallery YAML

It is inspired by LocalAI. It is MIT licensed. There is no telemetry, no analytics, and no account.

What works today

This repo is early development (0.1). The engine and API are real. Some items from the original vision are still on the roadmap.

Area	Status
Chat completions + SSE streaming	Working (echo now; llama.cpp when `llama-server` + GGUF are installed)
Completions, embeddings, TTS, STT, images	Working via built-in echo backends; real backends need their binaries
OpenAI name aliases (`gpt-4o`, `whisper-1`, …)	Working
Web UI (light/dark, persist selected model)	Working
Model gallery + CLI install	Working
Docker Compose + Helm chart	Working
Optional API key + Prometheus `/metrics`	Working
Multimodal vision (LLaVA), OCR, video	Not implemented yet
Bundled llama.cpp inside the Docker image	Not implemented yet — you provide `llama-server`

Until you install a GGUF and llama-server, every chat model name (gpt-4o, gpt-3.5-turbo, echo) returns the built-in echo reply. That is expected.

Requirements

To run the API and Web UI (echo backends only):

Go 1.23+ or Docker
Windows, macOS, or Linux

To run a real local LLM:

llama-server on your PATH (or set backends.llamacpp.binary)
Disk space for a GGUF (Llama 3.2 1B Q4 is roughly 0.8–1 GB)

Optional real backends:

Capability	Binary
Speech-to-text	`whisper-cli` from whisper.cpp
Text-to-speech	`piper`
Images	`sd` from stable-diffusion.cpp

Quickstart

From source (Windows / macOS / Linux)

git clone https://github.com/imranbru99/openlocalai.git
cd openlocalai
go run ./cmd/openlocalai serve --config config.yaml

Or build a binary:

# Windows
go build -o openlocalai.exe ./cmd/openlocalai
.\openlocalai.exe serve --config config.yaml

# Linux / macOS
go build -o openlocalai ./cmd/openlocalai
./openlocalai serve --config config.yaml

Docker

docker compose up -d --build

First request

The server is live immediately. No model download is required.

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"gpt-3.5-turbo\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello!\"}]}"

On Windows PowerShell:

Invoke-RestMethod http://localhost:8080/v1/chat/completions -Method POST -ContentType 'application/json' -Body '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Hello!"}]}'

URL	Purpose
http://localhost:8080	Web UI
http://localhost:8080/v1	OpenAI-compatible API
http://localhost:8080/health	Liveness
http://localhost:8080/metrics	Prometheus metrics
http://localhost:8080/v1/models	Model list

Web UI

Open http://localhost:8080

Page	What it does
Chat	Streaming chat. Enter sends, Shift+Enter inserts a newline. The Model dropdown is saved in the browser (`localStorage`) and restored after reload.
Models	Installed models + gallery. Install downloads weights into `models/weights/` and writes a YAML under `models/configs/`.
Playground	Raw `POST /v1/chat/completions` JSON in, JSON out.

Light and dark themes: use the toggle at the bottom of the sidebar. The choice is saved.

The UI is static files in web/ and is also embedded in the binary.

CLI

openlocalai serve [--config path]     Start the API + Web UI (default config.yaml)
openlocalai models list               List installed + builtin + alias names
openlocalai models gallery            Print the curated catalog as JSON
openlocalai models install <name>     Download a gallery model
openlocalai version                   Print version
openlocalai help

models commands read OPENLOCALAI_CONFIG if set, otherwise config.yaml in the current directory.

Why chat replies look like “echo”

If you see a message that starts with:

Hello from OpenLocalAI. This response is served by the built-in echo backend…

the request succeeded. No GGUF is loaded. Names like gpt-4o and gpt-3.5-turbo are aliases so official OpenAI SDKs work without code changes. Out of the box they point at the built-in echo chat model.

That is not a failed GPT-4o call. There is no cloud model behind those names.

Default model

Two different “defaults” exist.

1. Web UI default (this browser)

Change the Model dropdown on Chat. OpenLocalAI stores olai-chat-model in localStorage. Reload keeps your last choice if that model still exists.

2. Server / SDK default (`gpt-3.5-turbo`, `gpt-4`, `gpt-4o`)

Those names resolve through YAML alias: lists.

Today models/configs/echo.yaml owns them:

name: echo
backend: echo
alias:
  - gpt-3.5-turbo
  - gpt-4
  - gpt-4o

To make a real model the default for SDKs and for those alias names:

Add the alias on the model you want:

name: llama3.2-1b-instruct
backend: llamacpp
file: Llama-3.2-1B-Instruct-Q4_K_M.gguf
alias:
  - gpt-3.5-turbo
  - gpt-4o

Remove those names from echo.yaml so echo does not steal them back.
Restart the server.

You can also skip aliases and call the real name: "model": "echo" or "model": "llama3.2-1b-instruct".

Install a real LLM

Install llama.cpp and put llama-server on your PATH, or set:

backends:
  llamacpp:
    binary: C:\path\to\llama-server.exe

Install a small CPU-friendly model:

# from a built binary
openlocalai models install llama3.2-1b-instruct

# or from source
go run ./cmd/openlocalai models install llama3.2-1b-instruct

Restart openlocalai serve.
Move the gpt-3.5-turbo alias onto that model (see Default model), or select llama3.2-1b-instruct in the UI.

OpenLocalAI starts a private llama-server on a free localhost port and proxies OpenAI chat/completions/embeddings to it.

Model gallery

Catalog file: models/gallery.yaml.

Name	Backend	Notes
`llama3.2-1b-instruct`	llamacpp	Smallest recommended chat model
`llama3.2-3b-instruct`	llamacpp	Better quality, more RAM
`phi-3.5-mini-instruct`	llamacpp	Microsoft Phi-3.5 Mini
`whisper-tiny`	whispercpp	Fast STT
`whisper-base`	whispercpp	Better STT

Install from the CLI or the Models page in the UI (POST /admin/gallery/install).

Downloads go to models/weights/. A matching YAML is written to models/configs/<name>.yaml. If the gallery entry includes a SHA-256, the file is checksummed.

Bring your own model

Drop a GGUF into models/weights/ and add models/configs/my-model.yaml:

name: my-llama
backend: llamacpp
description: Custom local chat model
file: my-model.Q4_K_M.gguf
context_size: 8192
gpu_layers: 0
template: llama3
lora: adapters/style.gguf   # optional
grammar: ""                 # optional GBNF, also accepted per request
alias:
  - gpt-3.5-turbo

Restart the server. The model appears on GET /v1/models.

API reference

OpenAI request/response shapes. Unknown JSON fields are ignored.

OpenAI routes

Method	Path	Notes
`GET`	`/v1/models`	Installed models + aliases
`POST`	`/v1/chat/completions`	Set `"stream": true` for SSE (`data: …` then `data: [DONE]`)
`POST`	`/v1/completions`	Legacy completions
`POST`	`/v1/embeddings`	`input` is a string or array of strings
`POST`	`/v1/audio/transcriptions`	`multipart/form-data` field `file`
`POST`	`/v1/audio/speech`	JSON `{model, input, voice, speed}` → audio bytes
`POST`	`/v1/images/generations`	JSON `{model, prompt, n, size}` → `b64_json`

Chat fields forwarded to llama.cpp when present: tools, tool_choice, response_format, grammar, seed, stop, temperature, top_p, max_tokens.

Built-in aliases

Alias	Default target (until you override YAML)
`gpt-3.5-turbo`, `gpt-4`, `gpt-4o`	First chat model (`echo`)
`text-embedding-ada-002`, `text-embedding-3-small`	First embed model (`echo-embed`)
`whisper-1`	First transcribe model (`echo-transcribe`)
`tts-1`	First speech model (`echo-speech`)
`dall-e-2`, `dall-e-3`	First image model (`echo-image`)

Admin / ops routes (OpenLocalAI extensions)

Method	Path	Body
`GET`	`/health`	—
`GET`	`/metrics`	Prometheus text
`GET`	`/admin/status`	Loaded models + backend names
`POST`	`/admin/models/load`	`{"model":"echo"}`
`POST`	`/admin/models/unload`	`{"model":"echo"}`
`GET`	`/admin/gallery`	Catalog
`POST`	`/admin/gallery/install`	`{"name":"llama3.2-1b-instruct"}`

/health and the Web UI do not require an API key. /v1/* and /admin/* do, if one is configured.

Chat example

{
  "model": "gpt-3.5-turbo",
  "stream": false,
  "messages": [
    {"role": "system", "content": "You are a local assistant."},
    {"role": "user", "content": "Hello!"}
  ]
}

SDK examples

Change only base_url / baseURL. Use any string for api_key when auth is off (not-needed is fine).

Python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

print(client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello!"}],
).choices[0].message.content)

Also in examples/python/quickstart.py.

Node.js

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8080/v1",
  apiKey: "not-needed",
});

const res = await client.chat.completions.create({
  model: "gpt-3.5-turbo",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(res.choices[0].message.content);

Also in examples/node/quickstart.mjs.

curl (embeddings / speech / images)

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"text-embedding-ada-002","input":"local rag"}'

curl http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{"model":"dall-e-2","prompt":"a copper lamp"}'

curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"tts-1","input":"hello"}' \
  --output speech.wav

Backends

Each model YAML sets backend:. The engine loads that backend on first use and evicts the least-recently used instance when models.max_loaded is reached (default 2).

`backend`	Capabilities	Needs extra binary?
`echo`	chat, complete	No
`echo-embed`	embeddings (deterministic, 384-d)	No
`echo-transcribe`	speech-to-text placeholder	No
`echo-speech`	WAV beep	No
`echo-image`	generated PNG	No
`llamacpp`	chat, complete, embeddings	`llama-server`
`whispercpp`	transcription	`whisper-cli`
`stablediffusion`	images	`sd`
`piper`	TTS	`piper`

To add a backend: implement backend.Backend / backend.Instance, backend.Register in init(), and import the package from internal/cli.

More detail: docs/backends.md.

Configuration

Primary file: config.yaml. Environment variables override YAML.

server:
  host: 0.0.0.0
  port: 8080
  cors: true
  web_dir: web
  max_request_body: 67108864   # 64 MiB

workers:
  default: 2

models:
  directory: models/weights
  configs: models/configs
  gallery: models/gallery.yaml
  max_loaded: 2
  idle_unload: 30m

backends:
  llamacpp:
    binary: llama-server
    extra_args: []
  whisper:
    binary: whisper-cli
  stablediffusion:
    binary: sd
  piper:
    binary: piper

auth:
  api_key: ""                  # empty = no auth

Environment variable	Meaning
`OPENLOCALAI_HOST`	Bind address
`OPENLOCALAI_PORT`	Port (default `8080`)
`OPENLOCALAI_API_KEY`	Require `Authorization: Bearer …`
`OPENLOCALAI_MODELS_DIR`	Weight directory
`OPENLOCALAI_LLAMA_SERVER_BIN`	Path to `llama-server`
`OPENLOCALAI_WEB_DIR`	Override embedded UI
`OPENLOCALAI_CONFIG`	Config path for `models` CLI commands

See docs/configuration.md.

Authentication

Leave auth.api_key empty for local use.

To require a key:

# Windows PowerShell
$env:OPENLOCALAI_API_KEY = "change-me"

# Linux / macOS
export OPENLOCALAI_API_KEY=change-me

Clients send Authorization: Bearer change-me. /health and the Web UI stay open.

GPU

OpenLocalAI does not call CUDA/ROCm/Metal itself. Acceleration comes from the backend binary.

On a model YAML:

gpu_layers: 99

0 (default) is CPU-only.

Platform	Typical llama.cpp build
NVIDIA	CUDA
AMD	ROCm or Vulkan
Apple	Metal
Other	Vulkan or CPU

NVIDIA Compose overlay:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d

The default Docker image is CPU-only. See docs/gpu.md.

Docker

docker compose up -d --build

Image builds the Go binary and serves the embedded UI.
./config.yaml is mounted read-only.
./models/weights and ./models/configs are mounted so installs persist.

docker compose down

Kubernetes

Minimal chart: deploy/helm/openlocalai.

helm install openlocalai ./deploy/helm/openlocalai

Values of note: image name/tag, service port 8080, optional PVC for weights (persistence.size, default 50Gi), env.OPENLOCALAI_API_KEY.

Reverse proxy

Examples:

examples/nginx.conf — disable buffering, long proxy_read_timeout for streams
examples/Caddyfile — flush_interval -1
examples/traefik.yml

Streaming chat needs an unbuffered proxy path.

Project layout

OpenLocalAI/
├── cmd/openlocalai/          CLI entrypoint
├── internal/
│   ├── api/                  OpenAI + admin HTTP handlers
│   ├── backend/              Interface + echo, llamacpp, whisper, sd, piper
│   ├── cli/                  serve / models commands
│   ├── config/               YAML + env
│   ├── gallery/              Download + checksum
│   ├── metrics/              Prometheus text
│   ├── models/               Registry, aliases, LRU load/unload
│   └── server/               Middleware (CORS, auth, logs)
├── web/                      Chat / Models / Playground UI
├── models/
│   ├── gallery.yaml          Curated install catalog
│   ├── configs/              Per-model YAML
│   └── weights/              Downloaded GGUF / ggml files (gitignored)
├── deploy/helm/openlocalai   Kubernetes chart
├── examples/                 Python, Node, nginx, Caddy, Traefik
├── docs/                     Getting started, API, backends, GPU
├── config.yaml
├── docker-compose.yml
└── Dockerfile

Development

go test ./...
go run ./cmd/openlocalai serve --config config.yaml

Makefile targets: make build, make test, make run, make tidy, make docker.

CI: .github/workflows/ci.yml runs go test ./... and builds multi-platform release binaries.

Do not commit weights (.gguf, .bin, .safetensors).

Roadmap

See ROADMAP.md.

Multimodal chat (LLaVA / Moondream vision image inputs)
Real JSON-mode & structured output support
Real-time download progress tracking & job management
Cross-platform release binaries (Windows, Linux, macOS)
Automatic backend discovery & plugin expansion

Author & Maintainer

OpenLocalAI is designed, developed, and maintained exclusively by Imran Ahmed.

License

Acknowledgements

Inspired by LocalAI and the local-first AI community: llama.cpp, whisper.cpp, stable-diffusion.cpp, and Piper.

Author

Let's Build Something Exceptional

I'm actively open to:

Remote Senior Full-Stack Roles · Freelance Contracts · Technical Partnerships · Long-Term Collaborations

in Laravel · WordPress · React/Next.js · AI-powered Platforms · Security Audits · SaaS Architecture

📍 Timezone: UTC+6 (Dhaka/Rangpur) — flexible overlap for US, EU & Asia
⚡ Available: Immediately · Production-first · Fast delivery · Transparent communication

Platform	Link
🌐 Portfolio	imrandev.bd
💼 LinkedIn	linkedin.com/in/imranbru99
🐙 GitHub	github.com/imranbru99
📦 This repo	github.com/imranbru99/openlocalai
🐦 X / Twitter	@imrandev_bd
📺 YouTube	@ImranDevBD
📸 Instagram	@imranbru99
📘 Facebook	ExpertImranDev
🎵 TikTok	@imrandev_bd
🧵 Threads	@imranbru99
📌 Pinterest	@imrandev_bd
💬 WhatsApp	+880 1576-918420
📧 Email	me@imrandev.bd
🔗 All Links	linktr.ee/ExpertImranDev

"Security isn't an add-on — it's the foundation. Scale, speed, and trust drive every line of code I write."

— Imran Ahmed

Name		Name	Last commit message	Last commit date
cmd/openlocalai		cmd/openlocalai
deploy/helm/openlocalai		deploy/helm/openlocalai
docs		docs
examples		examples
internal		internal
models		models
web		web
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
ROADMAP.md		ROADMAP.md
config.yaml		config.yaml
docker-compose.gpu.yml		docker-compose.gpu.yml
docker-compose.yml		docker-compose.yml
go.mod		go.mod
go.sum		go.sum

Latest commit

History

Folders and files

Repository files navigation

OpenLocalAI

Contents

What it is

What works today

Requirements

Quickstart

From source (Windows / macOS / Linux)

Docker

First request

Web UI

CLI

Why chat replies look like “echo”

Default model

1. Web UI default (this browser)

2. Server / SDK default (gpt-3.5-turbo, gpt-4, gpt-4o)

Install a real LLM

Model gallery

Bring your own model

API reference

OpenAI routes

Built-in aliases

Admin / ops routes (OpenLocalAI extensions)

Chat example

SDK examples

Python

Node.js

curl (embeddings / speech / images)

Backends

Configuration

Authentication

GPU

Docker

Kubernetes

Reverse proxy

Project layout

Development

Roadmap

Author & Maintainer

License

Acknowledgements

Author

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

2. Server / SDK default (`gpt-3.5-turbo`, `gpt-4`, `gpt-4o`)