llms-txt Generator

Custom tooling for generating llms.txt documentation with DSPy. The project bootstraps a DSPy-based analysis pipeline that inspects repositories and produces LLM-friendly docs.

Getting Started

1. Clone the repository and switch to the dev branch.

2. Ensure you have Python 3.11+ available. A local virtual environment is checked in at .venv/ (ignored by git).

3. Automatic activation:

   • The repo ships with an .envrc for direnv. Run direnv allow once from the project root to auto-load .venv whenever you cd into the directory.

4. Install dependencies (recommended):

   uv sync

   This will install the project and all dependencies into .venv using the pyproject.toml lock metadata.

   Alternatively, you can fall back to pip:

   pip install -r requirements.txt

   or use the packaging metadata:

   pip install -e .

5. (Optional) Install development extras:

   pip install -e .[dev]

Project Layout

• src/llms_txt/: Python package for the DSPy pipeline (modules will be added in subsequent tasks).
• src/llms_txt/data_sources.py: Helpers to gather repository metadata locally or via GitHub.
• src/llms_txt/pipeline.py: DSPy signatures and RepositoryAnalyzer module that compose llms.txt content.
• src/llms_txt/cli.py: Typer application that runs the analyzer and writes llms.txt.
• .docs/: Internal documentation (e.g., dspy_api.md).
• .beads/: Lightweight task tracker database (llm-* issues).

CLI Usage

After installing dependencies, generate documentation via the Typer CLI:

uv run llms-txt --repo-path .

This scans the current directory, configures a DSPy LM (default openai/gpt-4o-mini), and writes llms.txt at the repository root.

For remote GitHub analysis:

uv run llms-txt --repo-url https://github.com/owner/repo --github-token $GITHUB_TOKEN

Set OPENAI_API_KEY (or pass --api-key) so DSPy can reach your language model. Use --dry-run to print output without writing, and --output PATH to customize the destination file.

Verifying Locally

• Preview the output without touching disk:

  uv run llms-txt --repo-path . --dry-run

• Regenerate llms.txt after changes:

  uv run llms-txt --repo-path .

  The file lives at the repo root and is ignored by git; inspect it manually (e.g., less llms.txt) to confirm the sections look correct.

Next Steps

Follow the Beads issues (starting with llm-4 and llm-5) to extend the CLI, document advanced configuration, and validate generated output. Future updates can add quality evaluation, multi-repo batching, or hosted surfaces.