experiment-framework

A reproducible experiment framework for software engineering research. Manages experimental subjects as Git submodules and verifies baseline correctness — all through a single CLI.

Prerequisites

• Python 3.12+
• uv
• Git

Setup

git clone <this-repo>
cd experiment-framework
uv sync

Quick Start

# Add a subject (you must know its test and install commands)
uv run experiments add https://github.com/pallets/click \
  --test-command "uv run --locked --no-default-groups --group dev tox run" \
  --install-command "uv sync"

# Verify everything passes
uv run experiments baseline

Output:

── Processing click ──
  Install: uv sync
  Test:    uv run --locked --no-default-groups --group dev tox run
  Installing dependencies...
  Running tests...

── Processing ultrajson ──
  Install: uv sync && uv pip install pytest
  Test:    uv run python -m pytest
  Installing dependencies...
  Running tests...

====================================
Baseline Verification

✓ click
✓ ultrajson

2 passed
0 failed
====================================

How It Works

Each subject is a Git submodule inside subjects/. You provide the exact shell commands for installing dependencies and running tests. There is no auto-discovery — you must know how the project is built and tested.

Architecture

experiment-framework/
├── configs/subjects.yml          # Subject metadata: name, path, commit, test_command, install_command
├── subjects/                     # Git submodules directory (one per subject)
├── experiments/
│   ├── cli.py                    # CLI entry point (click)
│   ├── models.py                 # SubjectConfig, SubjectMap
│   ├── git.py                    # Submodule CRUD: add, init, update, checkout
│   ├── installer.py              # Runs install commands in isolated environments
│   └── baseline.py               # Orchestrates: init → checkout → install → test → report
├── pyproject.toml
└── README.md

Environment Isolation

Each subject runs in its own isolated environment. The framework's .venv is explicitly unset in subprocesses so tools like uv and poetry resolve project-local environments, not the framework's own.

Commands

uv run experiments baseline

Verify every subject passes its test suite:

1. Initialize any missing submodules
2. Checkout the pinned commit
3. Install dependencies in an isolated environment
4. Run the test suite (stdout/stderr streamed to terminal)
5. Continue to next subject even if this one fails
6. Print a summary of passes and failures

uv run experiments baseline

uv run experiments add <repo-url>

Register a new subject as a Git submodule. Both --test-command and --install-command are required.

uv run experiments add https://github.com/pallets/click \
  --test-command "uv run --locked --no-default-groups --group dev tox run" \
  --install-command "uv sync"

Options:

Option	Short	Required	Description
--test-command	-t	yes	Shell command to run tests
--install-command	-i	yes	Shell command to install dependencies
--name	-n	no	Subject name (inferred from URL if omitted)

uv run experiments update

Fetch the latest commit for every submodule and update subjects.yml:

uv run experiments update

uv run experiments list

List all registered subjects with their pinned commits and commands:

uv run experiments list

Configuration (configs/subjects.yml)

Every subject requires name, path, test_command, and install_command. commit is optional but recommended for reproducibility.

subjects:
  - name: click
    path: subjects/click
    commit: 679a7a0eccbdded7a6e85680bdaaf08003765e01
    test_command: uv run --locked --no-default-groups --group dev tox run
    install_command: uv sync

  - name: ultrajson
    path: subjects/ultrajson
    commit: 6f60807ae2da2ba5a6b6449e78b1033591eb4aa0
    test_command: uv run python -m pytest
    install_command: uv sync && uv pip install pytest

Fields

Field	Required	Purpose
name	yes	Short identifier for the subject
path	yes	Relative path from repo root to the submodule
test_command	yes	Shell command that runs the project's test suite
install_command	yes	Shell command that installs the project and its dependencies
commit	no	Pinned commit hash for reproducibility

Determining Test and Install Commands

When adding a new subject, look at their CI configuration to find the right commands:

1. Open .github/workflows/ in the project's repository
2. Find the main test workflow (usually tests.yml or ci.yml)
3. Identify the install step that precedes the test step
4. Copy those exact commands

For example, if a project's CI does:

- run: uv sync
- run: uv run pytest

Then:

install_command: uv sync
test_command: uv run pytest

Adding a Subject Manually

You can add entries directly to subjects.yml without using the CLI:

1. Add a submodule manually:

   git submodule add https://github.com/example/project subjects/project

2. Add the entry to configs/subjects.yml:

   - name: project
     path: subjects/project
     commit: abc1234
     test_command: pytest
     install_command: pip install -e ".[test]"

Design Principles

• Explicit — test and install commands are always provided by the researcher; nothing is guessed
• Lightweight — no CI parsing, no file scanning, no auto-discovery
• Isolation — each subject runs in its own environment, no cross-contamination
• Resilience — one failing subject doesn't stop the rest
• Extensibility — research tools (PseudoSnake) can be inserted after baseline verification