Private LLM Stack

Self-hosted LLM infrastructure using Docker, Portainer and Open WebUI for private, web-accessible AI workloads.

This repository documents the architecture and deployment of a self-hosted large language model platform — a privately-run alternative to hosted AI chat products, with the model runtime, chat data, and infrastructure kept on a server I operate myself.

Note: This repo documents how the environment is architected and deployed. It does not include the LLM/model weights themselves (pulled at runtime via Ollama), and no real domains, IPs, hostnames, or credentials appear anywhere below. Ports shown are illustrative examples, not the actual values in use.

Overview

The goal was a private LLM backend, on hardware I control, that serves two kinds of consumers: a chat interface for interactive use, and direct API access for my own applications to call programmatically. Rather than building a single-purpose chatbot, Ollama is run as a shared internal inference service:

• Ollama, running natively on the host, loads and serves open-weight models over its HTTP API.
• Open WebUI, running in Docker, is one consumer of that API — it provides the interactive chat interface.
• Custom internal applications are another consumer — they call Ollama's API directly, bypassing Open WebUI entirely, to use the same models programmatically.
• Portainer, running in Docker, gives visibility into and control over the containerized side of the stack.

Architecture

              Internal network (not exposed to the public internet)
                                  │
        ┌─────────────────────────┼─────────────────────────┐
        │                         │                          │
   Open WebUI            Custom internal apps            Portainer
 (Docker container)       (direct API clients)         (Docker container)
        │                         │
        └────────────┬────────────┘
                      ▼
             Ollama (native systemd service)
                      │
            ┌─────────┴─────────┐
      qwen2.5-coder:14b     qwen3:14b

Ollama isn't containerized — it runs as a native systemd service alongside Docker on the same host, serving both Open WebUI and other internal applications directly over its HTTP API. Open WebUI and Portainer are each published on their own host port for access within the internal environment.

Technology Stack

Layer	Technology	Purpose
Host OS	Linux	Runs Docker and the native Ollama service
LLM runtime	Ollama (native install, systemd service)	Loads and serves an open-weight model over a local HTTP API
Chat interface	Open WebUI (Docker container)	Web UI for chatting with the model; handles user accounts, chat history, file uploads
Container management	Portainer CE (Docker container)	Visibility into container health/logs/resource usage
Models served	qwen2.5-coder:14b, qwen3:14b	Coder model for code-generation/assistance workloads; general-purpose model for broader chat and reasoning tasks

Service Architecture

Not everything here is a container — that's a deliberate thing to call out rather than gloss over:

• ollama — installed natively via Ollama's official install script, running as a systemd service (ollama.service) directly on the host, reachable by both the open-webui container and other internal applications.
• open-webui — a single Docker container (ghcr.io/open-webui/open-webui:main), started standalone (docker run) rather than as a Portainer/Compose stack, with its port published to the host.
• portainer — a standalone Docker container (portainer/portainer-ce), with its web UI published to the host, used to manage and monitor open-webui.

Open WebUI's connection to Ollama is configured through Open WebUI's own Admin Settings → Connections screen after first-run setup, pointing it at Ollama's address on the host.

Networking

• open-webui runs on Docker's default bridge network.
• Because Ollama runs on the host rather than in a container, open-webui reaches it over the network via the Docker bridge gateway to the host, rather than Docker's internal service-name DNS.

Persistent Storage

• Open WebUI: chat history, user accounts, and uploaded files persist in its backend data directory, mounted into the container.
• Ollama: downloaded model weights persist under Ollama's default local model storage location on the host filesystem (the standard path used by its native Linux installer).

Model Management

Models are pulled and managed directly through Ollama — either via ollama pull on the host, or through Open WebUI's model-management UI, which proxies model pulls to Ollama's API. There's no automated/scripted model provisioning; it's a manual, on-demand step.

Two models are currently installed:

Model	Used for
qwen2.5-coder:14b	Code generation and coding-assistance workloads, called directly from internal apps
qwen3:14b	General-purpose chat and reasoning, used both interactively via Open WebUI and programmatically

API Usage

Open WebUI isn't the only consumer of Ollama — internal applications call Ollama's HTTP API directly, using it as a shared local inference backend rather than routing everything through the chat UI. Ollama exposes both a native API and an OpenAI-compatible API, so client code can use either its own client libraries or any standard OpenAI-SDK-compatible client pointed at the local endpoint.

Current integration checks connectivity and available models via Ollama's native /api/tags endpoint:

curl http://<ollama-host>:<ollama-port>/api/tags

which returns the installed models (qwen2.5-coder:14b, qwen3:14b) available for an app to call. Generation itself goes through the chat endpoints:

Native API:

curl http://<ollama-host>:<ollama-port>/api/chat -d '{
  "model": "qwen2.5-coder:14b",
  "messages": [
    { "role": "user", "content": "Write a Python function that reverses a linked list." }
  ],
  "stream": false
}'

OpenAI-compatible API:

curl http://<ollama-host>:<ollama-port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3:14b",
    "messages": [{ "role": "user", "content": "Summarize this internal document." }]
  }'

<ollama-host> and <ollama-port> are placeholders — actual values aren't published here since the service is restricted to the internal network. Because these calls go straight to Ollama, they bypass Open WebUI entirely; the chat UI and internal apps are two independent consumers of the same backend, not layered on top of one another.

Deployment Process

As actually performed, in order:

1. Install Docker on the Linux host.
2. Install Ollama natively using its official Linux install script, which sets it up as a systemd service.
3. Run Open WebUI as a standalone container (docker run ... -p <host-port>:8080 ghcr.io/open-webui/open-webui:main), publishing its port to the host.
4. First-run setup: visit the published Open WebUI port, create the first account (becomes admin), then set the Ollama connection under Admin Settings → Connections.
5. Run Portainer as a standalone container for container visibility/management, with its UI published to the host.
6. Pull a model via Ollama or Open WebUI's UI.

There's no docker-compose.yml in this repo driving the deployment — Open WebUI and Portainer were each started with individual docker run commands rather than a Compose/stack definition, so nothing here is "deploy this repo and get the environment" — it's a description of manual steps taken directly on the host.

Monitoring

Portainer's dashboard provides container-level visibility — health status, resource usage, and live log streaming for open-webui — without a separate monitoring stack. Ollama, being a native systemd service rather than a container, is monitored separately via systemctl status / journalctl rather than through Portainer.

Screenshots

Screenshots of the live deployment (Open WebUI chat interface, Portainer's container list) are intentionally omitted from this public repository to avoid exposing real ports, hostnames, or internal configuration. (Add sanitized/redacted screenshots here if you'd like to showcase the running UI.)

Lessons Learned

• Running the LLM runtime natively rather than in a container simplifies host-level integration (systemd management, direct model storage) at the cost of Docker's default per-service network isolation — a trade-off worth making deliberately for any component meant to be reached by multiple internal consumers.
• Designing Ollama as a shared backend from the start — rather than building a chat-only deployment and retrofitting API access later — made it straightforward to add internal-app integrations: Open WebUI and custom apps are just two independent clients of the same service.
• Ad hoc docker run deployment is fast to stand up but harder to reproduce. Migrating Open WebUI and Portainer into a version-controlled Compose/stack definition is a natural next step for making the environment rebuildable from scratch.
• Writing infrastructure documentation is a good forcing function for revisiting assumptions — it's a useful periodic exercise to confirm the deployed configuration still matches intent as a project evolves.