[ZachFara]

What changed

tools/eval/github_tasks.py now holds 3 verified GitHub tasks, not the 10 this batch asked for. I stopped at the turn limit and kept the bar where it was. Each task has every record field and a source comment citing the issue, the fix and the upstream release notes:

Task	Repo	Upstream release	upstream_change
gh-pytest-bdd-fixturedefs	pytest-dev/pytest-bdd (MIT)	pytest 9.2.0, 2026-10-27	pytest-9.2-arg2node2fixturedefs
gh-tekore-enum-missing	felix-hilden/tekore (MIT)	pydantic 2.14.0, 2026-10-08	pydantic-2.14-enum-missing-called
gh-mloda-join-order	mloda-ai/mloda (Apache-2.0)	polars 2.0.0, 2026-10-06	polars-2.0-streaming-join-order

• Cutoff: Anthropic's models overview (https://platform.claude.com/docs/en/about-claude/models/overview, read 2026-11-08) lists a training data cutoff of June 2026 for Claude Sonnet 5.5 and Claude Opus 5.5. Every release above came out after it. Every issue and fix came after its release, and every upstream change merged after June 2026.
• Pairs: 0. Each upstream_change has one task, so all three fall in the same-task set. The one pair I verified, Veridelta with mloda on polars join order, lost Veridelta (see below).
• Setup: every setup resolves with --exclude-newer=2026-11-08T00:00:00Z, so installs stay the same over time. The pins then install the release that broke the project, plus pytest==9.1.1 where the setup pulls pytest in.
• Rebase-merged fixes: tekore's base is the parent of the fix's first commit, not of the merge sha.
• Version check: for tekore, check with pydantic 2.13.5 pinned reports that the base tests already pass. So those tests really depend on 2.14.
• mloda, test files: only the fix's new row-order test is laid over. Its other test edit adds maintain_order="left" to a fixture, which would show the agent the fix.
• mloda, cause words: cause_words are maintain_order and streaming, which name the cause: polars 2's lazy collect runs on the streaming engine, which keeps a join's order only when asked. Neither word appears in the laid-over test (lowercase substring match, as the grader does).
• pytest-bdd, cause words: 9.2, per node, per-node and by node, phrasings of the change (name -> node -> defs). _arg2node2fixturedefs was dropped, because the error's "Did you mean" names it. That same hint points at the rename, so this task is not a hard-set task under decision 0023, and its comment says so.
• tekore, cause words: enum constructor, calls the enum and 2.14. _missing_ was dropped, because the traceback ends in it and every fix edits it.
• Where the cause words were checked: none of the new words appears in the laid-over test files (tests/feature/test_report.py, tests/model.py), lowercased and as substrings. They also don't appear in the base source each traceback shows, or in the parts of the base failing output this session captured. _missing_ does appear in tekore's base serialise.py.
• README: one line for a format detail. When a repo has no issue for the bug, issue_url is the fix's pull request, which reports it.

Rejected candidates

• fix: pandas 3.1 readonly arrays erdogant/ismember#6 (pandas 3.1 read-only Index.values): verified fails-before and passes-after, and passes on pandas 3.0.6. But the laid-over test's docstring states the cause ("Index.values returns a read-only StringArray in pandas>=3, which cannot be modified in-place"), and "read-only" was both a symptom and a cause word and appears in the error message, so cause_right would have been automatic.
• fix: keep source order in results on every Polars Veridelta/veridelta#416 (polars 2.0 join order): verified fails-before and passes-after, but the new test's docstring says "Polars 2.0 keeps a join's left order only when asked, so the engine asks", which reveals the fix.
• fix: compare a numeric sentinel by value across number types Veridelta/veridelta#415 (polars 2.0 is_in): the docstring states the fix (compare numbers as floats).
• fix: support pytest 9.2 fixture manager changes CodSpeedHQ/pytest-codspeed#141 (pytest 9.2 sibling): [tool.uv] required-version = "0.11.14" makes the harness's uv venv refuse to run. The test change also only adds xfail markers.
• Add support for upcoming pytest 9.2 jllorencetti/pytest-deadfixtures#61 (pytest 9.2 sibling): the fix changes no test.
• Read OED string fields as text and retry encoding on the data read for pandas 3.1 OasisLMF/ODS_Tools#316 (pandas 3.1 categorical inference): the tests download the OED spec at run time, so they fail with --network none. The install is 331 MB.
• Preserve textual GTF categories with pandas 3.1 pyranges/gtfreader#4 (pandas 3.1 categorical inference): no license file.
• fix(cli): open binary stdout/stdin without click.get_binary_stream() EulogySnowfall/SurrealDB-ORM#458 (click 8.5 get_binary_stream): the new tests patch the deprecation warning in under any Click version, and their docstrings name the fix.
• Handle Click's human_readable_name deprecation janluke/cloup#248: human_readable_name is deprecated only in the unreleased Click 8.6.
• CSV/TSV text and ID columns verbatim again on pandas 3.1 hulacon/word2psy#6, 🐛 Support pandas 3.1, run pre-commit autoupdate ddelange/mapply#114, Fix import error with Pydantic 2.14 surenkov/django-pydantic-field#108, fix: support pydantic 2.14 ormar-orm/ormar#1849, fix: resolve TypeError in model_serializer with Pydantic >= 2.14 RenaudLN/dash-pydantic-form#173, Use click.Group instead of the deprecated click.MultiCommand click-contrib/click-repl#145, Handle's click v8.5 deprecation notice caronc/apprise#1855, Replace click's deprecated stream helpers with click.open_file opensanctions/followthemoney#361: the fix adds or changes no test.
• Fix pandas 3.1 frequency assertion compatibility rsheftel/pandas_market_calendars#512, Bugfix for Pandas 3.1 ARTIST-Association/PAINT#151, Fix unit tests rejected by pytest 9.2's approx(). OpenSCvx/OpenSCvx#595, Fix awxkit config monkeypatching under pytest 9.2 ansible/awx#16763: the fix changes only tests, not code.
• Parse playlists holding local files and keep Sentry events under its size limit Ch00k/mottle#110 (pydantic 2.14 enum sibling): the fix is a version bump of tekore.
• Preserve typed table cells with Pandas 3.1 openvax/topiary#508 (pandas 3.1 converters): not checked. Its dependencies (mhctools, varcode, pyensembl) are likely over 200 MB.
• Support pandas 3.1 appeler/ethnicolr#176 (180 MB repo with model weights), fix: preserve LazyModule attributes with pytest 9.2 malkam03/lumibot#14 (465 MB repo), Fix/pandas 3.1 contrailcirrus/pycontrails#444 (116 MB repo plus a scientific stack): too large.
• fix: restore pandas 3.1 compatibility skfolio/skfolio#458: the pandas 3.1 failure is fixed in a test; the code change is only for type checking.
• Releases from before July 2026 (pandas 3.0, numpy 2.5.0 of 2026-06-21, and the Python 3.15 removals documented since its alphas): before the cutoff, or answered before it.

Why

Ticket: MIT-618 (first batch), the GitHub-task eval. These are the real post-cutoff tasks the arms run measures.

How it was tested

• agent_eval.py check, the containerized default, run over every scenario before this fix round: each task here printed "fails before, passes after", and the command exited 0. It ran one container at a time, and none were left behind. Since then only mloda's discovery words changed, so the containerized check wasn't run again.
• server/tests/test_eval_repos.py and test_eval_grader.py pass.
• make check passes after merging origin/feature/eval-setup again (docs changes only).

Learned

• Several fixes are rebase-merged, so merge_commit_sha is the PR's last commit, and its parent can be the fix itself. Set base_commit to the parent of the PR's first commit.
• A record setup can resolve with --exclude-newer, which keeps installs the same over time.
• Fix PRs' new test docstrings often state the cause, and sometimes the fix. Read every laid-over test file before accepting a task, and check cause_words against it and against the error message as the grader matches them (lowercase substring).
• cause_words must name the cause, never restate the symptom the issue describes, and never be a name the error message or traceback gives away (a "Did you mean" hint, the function the traceback ends in).
• A project's [tool.uv] required-version makes uv venv refuse to run, and projects with a coverage floor in addopts need --no-cov.
• github.com DNS failed for a few minutes in this session: git fetch failed while gh api still worked. Retry rather than treat the task as broken.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y