CTFusion

A CTF-based Benchmark for LLM Agent Evaluation

CTFusion is a streaming evaluation framework that benchmarks large language model (LLM) agents on live Capture-the-Flag (CTF) competitions. Instead of relying on static, reusable challenge sets, CTFusion runs agents against active CTF events and measures how well they perform under realistic, contamination-resistant conditions.

ICML 2026 Workshop — AIWILD. This work appears at the Second Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD) at ICML 2026. Workshop homepage: AIWILD @ ICML 2026

---

Overview

Existing CTF benchmarks can be unreliable for evaluating LLM agents: published challenges are prone to data contamination, and agents can cheat (for example, by recovering known solutions through web search). CTFusion mitigates these problems by evaluating agents on live events through four mechanisms:

• Live CTF integration — agents are evaluated on active CTF events rather than static, reusable challenges.
• CTFd MCP server — a Model Context Protocol (MCP) server built on the widely used CTFd platform that exposes challenge listing, details, file download, and flag submission to agents.
• Per-agent independence — individual agents are isolated and their results tracked separately, even when they share a single team account.
• Competition impact reduction — only the first correct flag per challenge is forwarded to the live scoreboard. Repeat or incorrect submissions are answered from a local cache, so the benchmark does not disturb the ongoing competition.

The framework ships with two LLM agent architectures (D-CIPHER and Enigma) and supports OpenAI, Anthropic, and Google Gemini models.

Architecture

Component	Description
CTFd-MCP	FastAPI server (port 8000) that exposes CTFd challenges and flag submission to agents.
MCP-proxy	Flask orchestrator and web UI (port 5000). Launches agents, monitors runs, tracks per-agent results in SQLite, and de-duplicates flag submissions (the "first correct flag" logic).
EVAL-D-CIPHER	Multi-agent planner/executor CTF solver (fork of NYU D-CIPHER).
EVAL-Enigma	SWE-agent–based single-agent CTF solver (fork of Enigma).

Flag/challenge flow

agent (Docker container)  →  CTFd-MCP (:8000)  →  MCP-proxy (:5000)  →  live CTFd

Agents fetch challenges and submit flags through CTFd-MCP; the proxy intercepts submissions, forwards only the first correct flag per challenge to the live CTFd, and records the result.

Requirements

• Python 3.10+
• Docker (running)
• A CTFd instance and an API access token
• At least one LLM API key (OpenAI, Anthropic, and/or Google Gemini)

Installation and Setup

1. Clone the repository

git clone https://github.com/kaist-hacking/CTFusion.git
cd CTFusion

2. Configure environment variables

Copy the template and fill in your own values:

cp .env.example .env

.env holds your CTFd URL/token and LLM API keys. It is gitignored and must never be committed. See .env.example for the full list of variables and notes (for example, CTFD_BASE_URL must not include a trailing slash or the /api/v1 suffix).

3. Create and activate a virtual environment

python3 -m venv venv
source venv/bin/activate

4. Install dependencies and build images

bash setup.sh

setup.sh distributes the variables from .env to each component, installs the Python dependencies, builds the D-CIPHER Docker image (ctfenv:multiagent), pulls the Enigma image (sweagent/enigma:latest), and creates the ctfnet Docker network.

Running

Activate the virtual environment in each terminal before starting a server.

Run the CTFd-MCP server

python3 CTFd-MCP/server/server.py

Run the MCP-proxy server

In a separate terminal:

source venv/bin/activate
python3 MCP-proxy/app.py

Then open the web UI at http://localhost:5000, load the challenge list, and start a run for any challenge/model combination.

Project Structure

CTFusion/
├── .env.example          # Environment variable template (copy to .env)
├── setup.sh              # Setup: env distribution, deps, Docker images, network
├── CTFd-MCP/             # CTFd MCP server (FastAPI, :8000)
├── MCP-proxy/            # Orchestrator + web UI (Flask, :5000)
├── EVAL-D-CIPHER/        # D-CIPHER multi-agent solver
└── EVAL-Enigma/          # Enigma (SWE-agent) solver

Configuration Notes

• Models — the agent/model combinations are configured in MCP-proxy/app.py and the per-backend model registries (EVAL-D-CIPHER/.../backends/ and EVAL-Enigma/sweagent/agent/models.py). Make sure the model IDs you use are available to your API account.
• Cost budget — each D-CIPHER run is bounded by experiment.max_cost in the EVAL-D-CIPHER/configs/dcipher/*.yaml configs. Tune this to your model: more capable (and more expensive) models need a higher budget to complete a challenge.

Notes

• The .env file and the generated keys.cfg files contain secrets. They are gitignored; do not commit them.
• Rotate and remove unused API keys regularly.
• Press Ctrl+C in the respective terminal to stop each server.

Citation

If you use CTFusion in your research, please cite:

@misc{ctfusion2026,
  title        = {CTFusion: A CTF-based Benchmark for LLM Agent Evaluation},
  author       = {Dongjun Lee and Ga-eun Bae and Insu Yun},
  howpublished = {Accepted at the ICML 2026 Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD)},
  year         = {2026}
}

License

This project is released under the MIT License. See LICENSE for details.

The EVAL-D-CIPHER/ and EVAL-Enigma/ directories are derived from third-party projects and retain their own upstream licenses (see the LICENSE file inside each).

Acknowledgements

CTFusion builds on two open-source CTF-solving agents:

• EVAL-D-CIPHER — based on NYU-LLM-CTF/llm_ctf_automation (D-CIPHER), MIT License, © 2024 NYU Tandon School of Engineering and NYU Abu Dhabi.
• EVAL-Enigma — based on SWE-agent / Enigma, MIT License, © 2024 John Yang, Carlos E. Jimenez, Alexander Wettig, Shunyu Yao, Karthik Narasimhan, Ofir Press.