Agency

A command-line LLM agent in Rust. It streams responses from an OpenAI-compatible model server, calls tools on the model's behalf, remembers conversations, connects to MCP servers, and answers questions over your own documents.

Built step by step as a learning project — each capability is a milestone with its own runnable example.

Requirements

• Rust 2024 edition (stable toolchain).
• An OpenAI-compatible model server. LM Studio at http://localhost:1234/v1 is the default. Load a chat model before running.
• Optional: Node and npx, for the MCP example.
• Optional: an embedding model loaded in LM Studio, for retrieval.

Build

cargo build --release
cargo run -- --help

Getting started

Single-turn prompt:

cargo run -- --model "lfm2.5-2.6b-mlx@8bit" "Hello, what are you?"

Interactive REPL — omit the prompt:

cargo run -- --model "lfm2.5-2.6b-mlx@8bit"

Built-in tools and context-window trimming are active by default in both modes. In the REPL, every conversation is saved as it happens.

Configuration

Settings resolve in the order command-line flag → config file → default, so a flag always wins.

The config file lives at ~/.config/agency/config.toml (or $XDG_CONFIG_HOME/agency/config.toml), and lets you avoid repeating --model and --base-url:

[defaults]
provider = "local"

[providers.local]
base_url = "http://localhost:1234/v1"
default_model = "lfm2.5-2.6b-mlx@8bit"

[providers.remote]
base_url = "https://example.invalid/v1"
api_key = "..."
default_model = "some-model"

Then select a provider with --provider remote, or rely on [defaults] provider.

Command-line flags

Flag	Default	Description
-m, --model	from config	Model name as shown by the server
-p, --provider	from config	Named provider section to use
--base-url	http://localhost:1234/v1	OpenAI-compatible API base URL
--api-key	—	Bearer token for authenticated endpoints
-s, --system	—	System prompt
--config	~/.config/agency/config.toml	Alternative config file
--db	~/.local/share/agency/agency.db	Conversation database (REPL)
--no-tools	tools enabled	Disable the built-in tools
--shell-policy	ask-once	ask-once, always-allow, or always-deny
--context-size	16000	Token budget before history is trimmed
--strategy	sliding-window	sliding-window or summarise

A model must come from either --model or the config file; there is no built-in default.

Built-in tools

The model can call these unless --no-tools is given:

Tool	Description
read_file	Read a file's contents
list_dir	List a directory
run_shell	Run a shell command

run_shell is gated by --shell-policy. The default, ask-once, prompts for approval the first time each distinct command is requested and remembers the answer for the session.

REPL commands

Command	Description
/help	List commands
/attach <path>	Attach an image, audio file, or document to the next message
/clear	Clear history and pending attachments
/save [file]	Save the conversation to JSON (default conversation.json)
/load <file>	Load a conversation from JSON
/resume	List saved conversations
/resume <id>	Continue a saved conversation
/quit	Exit — as does Ctrl-D

Examples

Each example demonstrates one capability in isolation and reads AGENCY_* environment variables, so it runs without flags.

cargo run --example hello        # streaming a single completion
cargo run --example repl         # multi-turn conversation
cargo run --example config       # provider presets from the config file
cargo run --example vision       # send an image to a vision model
cargo run --example context      # context trimming strategies
cargo run --example tools        # built-in tools
cargo run --example persistence  # SQLite persistence and /resume
cargo run --example mcp          # tools from an MCP server (needs npx)
cargo run --example rag          # question answering over doc/ (needs embeddings)

For example, to point any of them at a specific model:

AGENCY_MODEL="lfm2.5-2.6b-mlx@8bit" cargo run --example tools

The RAG example accepts --mode explicit (the model calls a search tool) or --mode implicit (context is retrieved automatically each turn).

Documentation

• doc/architecture.md — how the code is structured
• doc/plan.md — remaining work
• doc/in-process-inference.md — design notes for running models in-process
• doc/working-practices.md — contribution conventions

License

MIT