Configured llama.cpp in a folder

get models

Models are saved in gguf in the models subfolder. Configuration and parameters are in models.ini.

Current list of files. Made with:

echo -e "# models directory structure\n\n\`\`\`text\n$(tree -h --du --dirsfirst -F models)\n\`\`\`" > modelsDirTree.md

Folders added with:

find models -type d ! -path '*/\.*' -print0 | xargs -0 -I {} sh -c 'echo "*.gguf" > "{}/.gitignore"'

run engine

ROCm, Vulkan and CPU docker containers configured in compose.yaml.

Run with:

docker compose up -d

Logs with:

docker logs llamacpp-r
#or
docker logs llamacpp-v
#or
docker logs llamacpp-c

use it

Built-in WebUI and all APIs are under http://localhost:8083/ for Vulkan, http://localhost:8080/ for ROCm or http://localhost:8082/ for CPU.