MAESTRO

Multi-Agent Evaluation for Structured Relational Output

Comparing agentic orchestration frameworks for automated relational diagram generation.

---

What it evaluates

MAESTRO is a benchmark. It gives every configuration the same task, a structured input dataset to turn into a relational Mermaid diagram, then scores the output against a ground-truth diagram. The question it answers is whether multi-agent orchestration produces better relational output than a single agent, and at what cost.

Four orchestration strategies generate the diagram, holding prompts and the output contract identical so only the orchestration differs:

• single_agent: one prompt, one LLM call (the baseline)
• sop_based: a hand-coded three-step procedure (extract entities, extract relationships, render Mermaid)
• crew_ai: the same three steps orchestrated with CrewAI
• lang_graph: the same three steps orchestrated with LangGraph

Three control conditions (no LLM, deterministic) bracket the score range so a strategy's numbers are interpretable: null_control (empty diagram) and copy_control (raw input) are the floor; ground_truth_control (the answer verbatim) is the ceiling.

Five providers are under test: Anthropic, OpenAI, Mistral, Gemini, and DeepSeek, across a matrix of inputs x strategies x models x repeats, stratified by complexity tier.

Scoring covers structural validity (does it parse, via mmdc), entity F1 (id / name / lemma), relationship F1 (relaxed / strict), and an error taxonomy of what each diagram got wrong. Every cell is repeated and variance is reported.

---

Quickstart

The benchmark runs a matrix of inputs x strategies x models x repeats, scores each generated Mermaid diagram against its ground truth, and records every result (plus the runtime environment) in a SQLite database. The four steps below are the minimum to get a scored row into the database from a clean checkout. For the full CLI reference, resume semantics, cost expectations, and troubleshooting, see docs/running.md.

Docker is the recommended way to run MAESTRO. Local install is supported for day-to-day development.

Prerequisites

• Docker (recommended), or Python 3.11 for the local path.
• API keys for the providers you intend to exercise: Anthropic, OpenAI, Mistral, Gemini, DeepSeek.
• mmdc for the structural-validity metric. Bundled in the Docker image; optional locally (the metric is skipped if absent).

1. Clone and install

git clone https://github.com/Colinho22/maestro.git
cd maestro

Docker (recommended), which bundles Python, mermaid-cli, and Chromium:

docker compose build

Or install locally:

pip install -e .            # add ".[dev]" for the test/lint tools

2. Configure API keys

cp .env.template .env       # then edit .env with your keys

3. Smoke run

Docker:

docker compose run --rm maestro python -m maestro.run \
  --strategy single_agent --tier 1 --repeats 1

Local:

python -m maestro.run --strategy single_agent --tier 1 --repeats 1

4. Analyse or explore

Docker:

docker compose up                    # dashboard at http://localhost:8501
docker compose run --rm maestro python -m maestro.analysis

Local:

python -m maestro.analysis          # scored summary to stdout
streamlit run src/maestro/viz/app.py

The full matrix, filter combinations, and troubleshooting live in docs/running.md. The provenance model and DB integrity verification live in docs/reproducibility.md.

---

Local development

Setup is tested on macOS. Install the dev extras and run the test suite and linters from the project root:

pip install -e ".[dev]"
pytest
ruff check .
ruff format --check .

pre-commit hooks (ruff lint + format) are configured in .pre-commit-config.yaml; enable them with pre-commit install.

---

Citing

If you use MAESTRO in your work, please cite the archived release on Zenodo (10.5281/zenodo.20792756, which always resolves to the latest version) or the CITATION.cff file (GitHub's "Cite this repository" button).