AgenttiX AI server configs

Structure of the setup as an SVG figure

Software stack

• GPT Researcher
• LiteLLM
  • Routes requests to multiple backends, including both cloud services and local models on multiple devices.
• LiteLLM Claude
  • Use Claude Pro or Max subscription instead of an API key
• llama.cpp
  • For running local models
  • RTX 3070 (8 GB) + EPYC config of agx-ai (Gemma 4 26B-A4B on the GPU and the CPU, Qwen3.8-Flash-Next with experts in RAM), with the GPU power limit and measurements
  • RTX 3090 config of agx-z2e, with KV cache, context size and MTP measurements, and Qwen3.8-Flash-Next with experts in RAM
  • Custom container for Radeon VII with ROCm, with measurements
  • Dual RTX A4000 configs for PaperQA2 RAG (Qwen3.8-27B, Gemma 4 26B-A4B on 2 GPUs or 1 GPU + CPU), with VRAM, context size and speed measurements
  • NVIDIA T550 Laptop GPU (4 GB) config for Docker Desktop on Windows
  • Scripts for running llama.cpp natively on Windows
  • ThinkPad L14 Gen 5 (Intel Core Ultra 5 125U) iGPU config
  • ThinkPad L14 Gen 5 NPU config with OpenVINO Model Server
  • ThinkPad L14 Gen 5 NPU config for Gemma 4 E4B with a third-party server
  • ThinkPad L14 Gen 5 NPU config for EmbeddingGemma 2 with OpenVINO Model Server and a converter
  • ThinkPad T480 (Intel Core i7-8550U, NVIDIA GeForce MX150 2 GB) CPU + dGPU config
  • Benchmarks: measure the served configuration over HTTP, or run llama bench in the container
• LM Studio
  • For running local models
• vLLM
  • For running local models
  • Radeon VII config with a patched gfx906 container and a bit-exact Gemma 4 QAT checkpoint, with a comparison to llama.cpp
  • RTX 3090 configs of agx-z2e (Qwen3.8-27B, Gemma 4 26B-A4B and 31B, Qwen3.8-Flash-Next with experts in RAM and its vision tower on the CPU), with a Triton attention fix for MTP and a comparison to llama.cpp
  • RTX 3070 (8 GB) configs of agx-ai (Gemma 4 26B-A4B and Qwen3.8-Flash-Next with their experts in RAM over UVA), with fixes for UVA weight offloading, a re-quantized Flash-Next checkpoint and a comparison to llama.cpp
  • Dual RTX A4000 configs for PaperQA2 RAG: 8 LLM and embedding model configurations on 1-2 GPUs and the CPU, an embedding model retrieval test, and a PaperQA2 comparison to llama.cpp
• Open Terminal
  • Docker container for LLM terminal access
• Open WebUI
  • User interface for chats
  • Android client: Conduit

Cloud providers

• Claude

Other software

• Claude Desktop for Linux

Notes

• Don't oversaturate the CPU with too many threads

Connecting LiteLLM to a remote llama.cpp server over SSH

This can be done using the SSH sidecar container in docker-compose.yml. First, create ./ssh/config/private.conf based on ./ssh/config/private-template.conf, and run chown root:root ./ssh/config/private.conf. Start the SSH container with cd ./ssh && sudo docker compose run --rm ssh-debug. Create an SSH key from within the container with ssh-keygen -t ed25519 -C "YOUR_COMMENT". Your public key is now in ./ssh/config/id_ed25519.pub. On the jump host, set in ~/.ssh/authorized_keys:

restrict,port-forwarding,command="/bin/false",permitlisten="127.0.0.1:1",permitopen="LLAMA_HOSTNAME:22" YOUR_PUBLIC_KEY

On the llama.cpp server, set in ~/.ssh/authorized_keys:

restrict,port-forwarding,command="/bin/false",permitlisten="127.0.0.1:1",permitopen="127.0.0.1:9931" YOUR_PUBLIC_KEY

Either populate ./ssh/config/known_hosts manually, or disable StrictHostKeyChecking yes in ./ssh/config and run ssh -N -F /ssh/config -L 0.0.0.0:9931:127.0.0.1:9931 remote.