Skip to content

Add LiteLLM proxy + cloudflared tunnel for vLLM at model.formulacode.org - #22

Merged
atharvas merged 1 commit into
mainfrom
feat/model-proxy
May 4, 2026
Merged

Add LiteLLM proxy + cloudflared tunnel for vLLM at model.formulacode.org#22
atharvas merged 1 commit into
mainfrom
feat/model-proxy

Conversation

@atharvas

@atharvas atharvas commented May 4, 2026

Copy link
Copy Markdown
Member

Summary

  • Expose local vLLM servers (127.0.0.1:8123, :8124) at https://model.formulacode.org via a new datasmith-model Cloudflare Tunnel feeding a LiteLLM proxy on 127.0.0.1:4100.
  • LiteLLM runs in DB-backed mode against a new litellm database in the local Supabase Postgres, enabling the admin UI at /ui/, virtual keys, and spend logs.
  • One Make target (make model-tunnel) starts both LiteLLM and cloudflared together; persistent venv at .venv-litellm/ is built once via make model-proxy-install.

Files

  • infra/litellm.config.yaml — model list (gemma-4-31b on 8123, model-b placeholder on 8124).
  • Makefilemodel-proxy-install (one-time venv + prisma generate) and model-tunnel (boot LiteLLM, wait for health, run cloudflared).
  • docs/guide/model-proxy.md — architecture, operator setup, virtual-key issuance, troubleshooting.
  • mkdocs.yml, CLAUDE.md — pointers to the new doc.
  • .gitignore — ignore .venv-litellm/.

Test plan

  • make model-proxy-install succeeds.
  • make model-tunnel boots; /health/liveliness returns 200.
  • POST /v1/chat/completions with model=gemma-4-31b round-trips through tunnel → LiteLLM → vLLM and returns expected output.
  • https://model.formulacode.org/ui/ returns 200 (no "Not connected to DB").
  • POST /key/generate mints a virtual key; POST /key/delete revokes it.
  • CI: make check + tests pass on Python 3.11 / 3.12.

@atharvas
atharvas merged commit 89efaad into main May 4, 2026
3 checks passed
@atharvas
atharvas deleted the feat/model-proxy branch May 4, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant