Issues

• Add more non-trivial benchmark tasks that fit in 15 minutes

  evanwtf/limerick-benchmark#34 · Filter by author evandhoffman opened on Apr 19, 2026
  1

• Tighten limerick evaluator to require a full 5-line limerick, not a first-line match

  severity:medium
  evanwtf/limerick-benchmark#32 · Filter by author evandhoffman opened on Apr 17, 2026

• Do not pre-bootstrap Aider workspaces with uv init or flask

  severity:medium
  evanwtf/limerick-benchmark#33 · Filter by author evandhoffman opened on Apr 17, 2026

• Aider: evaluate more tolerant edit-format strategies for weaker local models

  enhancementseverity:low
  evanwtf/limerick-benchmark#31 · Filter by author evandhoffman opened on Apr 17, 2026

• Runner: write a job-level rollup to results/<job_id>/job_summary.json

  severity:low
  evanwtf/limerick-benchmark#28 · Filter by author evandhoffman opened on Apr 17, 2026

• Report: suppress 'Aider stagnation timeout' row at default unless triggered

  severity:low
  evanwtf/limerick-benchmark#27 · Filter by author evandhoffman opened on Apr 17, 2026

• Report: include started_at timestamp in each model section

  severity:low
  evanwtf/limerick-benchmark#26 · Filter by author evandhoffman opened on Apr 17, 2026

• Report: link to trace.jsonl from each model section

  severity:low
  evanwtf/limerick-benchmark#25 · Filter by author evandhoffman opened on Apr 17, 2026

• Report: surface entry_point_candidates for failures and non-app.py passes

  severity:low
  evanwtf/limerick-benchmark#24 · Filter by author evandhoffman opened on Apr 17, 2026

• Report: suppress zero-valued agent telemetry for Aider runs

  severity:low
  evanwtf/limerick-benchmark#23 · Filter by author evandhoffman opened on Apr 17, 2026

• Report: replace 'Finish Reasons' aggregate with 'Failure Categories'

  severity:low
  evanwtf/limerick-benchmark#22 · Filter by author evandhoffman opened on Apr 17, 2026

• Report: show failure_category in the summary table

  severity:low
  evanwtf/limerick-benchmark#21 · Filter by author evandhoffman opened on Apr 17, 2026

• ANTHROPIC_API_KEY leaks into agent-spawned bash subprocesses

  bugseverity:high
  evanwtf/limerick-benchmark#8 · Filter by author evandhoffman opened on Apr 17, 2026