IntentSpec

Overview

IntentSpec introduces Intent Violation Rate (IVR): a metric measuring how often LLM-generated code passes the tests a developer explicitly stated while violating unstated developer intent captured by hidden gold-constraint tests.

This repository contains:

• A benchmark of 49 algorithm spec pairs derived from HumanEval+ (data/specs/spec_pairs.jsonl), each with:
  • an ambiguous prompt (what a developer might actually write),
  • a gold prompt (the fully-clarified version),
  • a stated test (C1), and
  • 1–3 hidden constraint tests (C2–C4) that only the gold prompt implies.
• Canonical reference solutions (data/specs/canonicals.jsonl) used to validate that the benchmark itself is sound (see Validating the Benchmark).
• A pipeline that generates candidate solutions with the Anthropic API, executes each one against the stated and hidden tests in a sandboxed subprocess, and computes IVR overall and by specification type.

IVR for a spec pair is the fraction of solutions that pass the stated test(s) yet fail at least one hidden constraint:

IVR = n_violating / n_passing_stated

Requirements & Installation

• Python 3.10 or 3.11

pip install -e ".[dev]"

This installs the intentspec package plus its runtime dependencies (anthropic, evalplus, pydantic, numpy, pandas, rich, matplotlib) and dev tools (pytest, ruff, mypy) declared in pyproject.toml.

API Key Setup

Generation requires ANTHROPIC_API_KEY to be set in your environment. scripts/run_experiment1.py will fail as soon as it needs to call the Anthropic API if this is not set. It is not required if you only run against the cached generations already committed to this repo (see below).

export ANTHROPIC_API_KEY=sk-ant-XXX

Repository Structure

data/
  specs/
    spec_pairs.jsonl          # 49 spec pairs (ambiguous/gold prompts + constraint tests)
    canonicals.jsonl          # canonical reference solutions, keyed by task_id
  generations/                # cached LLM solutions, one file per model/task_id
src/intentspec/
  schema.py                   # SpecPair, ConstraintTest, SolutionResult, IVRResult (pydantic)
  dataset.py                  # load_spec_pairs(), load_humaneval_plus()
  generate.py                 # generate_solutions() — calls Anthropic/OpenAI, caches to disk
  execute.py                  # run_solution(), evaluate_solution() — sandboxed subprocess exec
  ivr.py                      # compute_ivr(), compute_ivr_by_type()
scripts/
  run_experiment1.py          # main entry point: generate -> evaluate -> compute IVR
                               #   (MODEL constant selects claude-sonnet-4-6 vs gpt-4.1)
  validate_benchmark.py       # soundness check: canonical solutions vs. all constraint tests
  plot_ivr_distribution.py    # histogram of per-task IVR from a results file
  flatten_specs.py            # maintenance utility: dedupe spec_pairs.jsonl by task_id
compute_ci.py                  # 95% bootstrap CI on overall IVR, per model (see below)
results/
  experiment1.json            # full results from the last run_experiment1.py run (claude-sonnet-4-6)
  experiment1_gpt-4_1.json    # full results from run_experiment1.py with MODEL = "gpt-4.1"
  experiment1_ivr_distribution.png
  experiment1_gpt-4_1_ivr_distribution.png
tests/                        # unit tests for schema/dataset/execute/ivr

Reproducing the Results

pip install -e ".[dev]"
python scripts/run_experiment1.py

run_experiment1.py takes no arguments. It loads all 49 spec pairs from data/specs/spec_pairs.jsonl, generates 5 solutions per spec pair with claude-sonnet-4-6, evaluates each solution against its stated and hidden constraint tests, prints per-task and overall IVR, and writes full results to results/experiment1.json.

This repository ships with the cached generations used for the paper (data/generations/claude-sonnet-4-6__<task_id>.jsonl, 5 solutions per task). generate_solutions() checks this cache before calling the API: if a cache file already has ≥5 solutions for a task, generation is skipped entirely and the cached solutions are reused.

• Running against the committed cache (default) reproduces the paper's reported IVR of 54.5% exactly, and does not require ANTHROPIC_API_KEY.
• If you delete or empty data/generations/, generation will run fresh against the Anthropic API. Fresh generation calls the API at its default temperature (1.0, nondeterministic). A fresh run will not reproduce 54.5% exactly, since it draws a different sample of completions each time.

To force fresh generation for reproducibility experiments of your own:

rm -rf data/generations
export ANTHROPIC_API_KEY=sk-ant-...
python scripts/run_experiment1.py

Bootstrap Confidence Intervals

python compute_ci.py

Takes no arguments. For each of results/experiment1.json (Claude Sonnet 4.6) and results/experiment1_gpt-4_1.json (GPT-4.1), it recomputes overall IVR from the same qualifying problems run_experiment1.py used (only spec pairs with ≥1 stated-passing solution), then estimates a 95% CI by resampling those per-problem IVRs with replacement (10,000 resamples, same n as the number of qualifying problems) and taking the 2.5th/97.5th percentiles of the resulting means.

np.random.seed(42) is set once before the bootstrap loop for each model, so the CI bounds are deterministic and identical on every run — this is what makes the reported CIs reproducible in CI (and re-running it yourself should print exactly:

Claude Sonnet 4.6: IVR = 54.5% (95% CI: 40.4%–68.1%), n=47
GPT-4.1: IVR = 63.5% (95% CI: 50.0%–76.5%), n=46

compute_ci.py only reads results/*.json — it never modifies the result files it consumes.

Validating the Benchmark

python scripts/validate_benchmark.py

Takes no arguments. For every spec pair, it resolves a gold reference solution from gold_refs/<task_id>.py if present, otherwise from canonicals.jsonl (aliasing entry_point to solution), and runs it against every constraint test (stated and hidden). It confirms that correct, canonical solutions pass all constraints (IVR = 0 across all specs), which shows the hidden tests are satisfiable by correct code rather than being spuriously unsatisfiable or contradictory. It never modifies specs, test code, or canonicals.jsonl.

Expected Output

Running python scripts/run_experiment1.py against the committed cache prints one line per spec pair, e.g.:

[HumanEval/26] (algorithm) Generating 5 solutions...
  Solution 1: stated=PASS, hidden_failed=1/2 (['C2'])
  ...

--- IVR Results ---
  HumanEval/26: IVR=0.400 (2/5 passing-stated solutions violate >= 1 hidden constraint)
  ...

Overall IVR (mean across tasks with >= 1 passing-stated solution): 0.545

IVR by spec type:
  algorithm: 0.545

Full results saved to results/experiment1.json

• Overall IVR ≈ 0.545 (54.5%), averaged over the 47 of 49 spec pairs that had at least one solution pass the stated test. The other 2 (HumanEval/101, HumanEval/112) had zero solutions pass their stated test in any of the 5 samples and are excluded from the mean, since IVR is undefined without at least one stated-passing solution.
• Across all 245 generated solutions (49 problems × 5 samples), the stated test C1 passes 94.3% of the time (14 failures). Hidden constraints fail far more often: C2 42.9% (99/231), C3 28.9% (58/201), C4 15.4% (10/65) — computed over the solutions whose problem actually included that constraint (not every problem has a C3 or C4), matching the paper's Table 1.
• All 49 spec pairs are spec_type = "algorithm" in this release, so the by-type breakdown currently has a single entry equal to the overall IVR.
• results/experiment1.json records, per task and per solution, whether the stated test passed and which hidden constraint IDs (C2/C3/C4) passed or failed — this is what the per-constraint fail rates above are derived from.
• python scripts/plot_ivr_distribution.py [results/experiment1.json] (results file argument optional, defaults to results/experiment1.json) renders this as a histogram, saved as <name>_ivr_distribution.png next to the input file. Note this produces a PNG; the ivr_distribution.pdf figure embedded in the paper was exported separately and is not regenerated by this script as-is.

Citation

@misc{intentspec2026,
  title        = {IntentSpec: Measuring Intent Violation Rate in LLM Code Generation},
  author       = {Anote AI Research},
  year         = {2026},
  howpublished = {\url{https://github.com/anote-ai/research-intentspecification}},
  note         = {Preprint}
}