[ZachFara]

What changed

tools/eval/github_tasks.py gets 1 more verified GitHub task, for 6 in all. Zach wants 20, so this is well short. I stopped at the turn budget and kept the bar where it was. The other leads in the brief all failed it (see the rejected candidates).

Task	Repo (license)	Upstream release	upstream_change
gh-marklassian-link-labels	mbroton/marklassian-py (MIT, 349 KB)	mistune 3.4.0, 2026-10-19	mistune-3.4-label-parsed-first

• Pairs: 0. The new key has one task.
• Timing:
  • The upstream change is lepture/mistune@bde6ad7 ("Refactor link and emphasis parsing"), authored 2026-07-22 and committed 2026-08-21, both after the June 2026 cutoff.
  • Fix feat: support mistune 3.4 mbroton/marklassian-py#26 merged 2026-10-30.
  • There is no issue for it, so issue_url is the PR, as the README allows.
• Merge commit: base_commit is the merge's first parent, 3ebadd3.
• Setup: uv pip install --exclude-newer=... -e . pytest==9.1.1 jsonschema, then the mistune==3.4.0 pin. The base caps mistune below 3.4, and the pin installs over that cap. The install is a few MB.
• Fix: it changes code. The mention plugin no longer reads state.in_link or state.in_image. Instead, the converter decides link and image context from the token tree and link marks.
• Hard-set: not claimed. The error doesn't name its own fix, but I didn't test whether the obvious fix fails.

Base failing output (containerized)

tests/test_mentions.py, the whole file. At the fix sha it has 26 tests.

Run	Result
Base, mistune 3.4.0	6 failed, 20 passed
Fix, mistune 3.4.0	26 passed
Base, mistune 3.3.4	26 passed (previous version, checked)

• The six that fail at the base are the fix's new test test_jira_mention_in_formatted_link_label_keeps_all_marks, plus five existing link and image-alt mention tests.
• Key line:

  E         Left contains one more item: {'type': 'mention', 'attrs': {'id': 'linked-id'}}
  

• The run went through a .scratch/ helper that calls repos.materialize, repos.prepare and repos.run_tests with container.run, the same steps agent_eval.py check runs. Containers ran one at a time, none were left behind, and --on-host was never used.

Cause-word checks

• Cause words:
  • parses link labels
  • parses link and image labels
  • before it knows
  • new link parser
  • 3.4
• Dropped on review: labels before, label before, parsed before and parses the label. A fix that only treats the symptom says them too ("strip mentions from the label before building the link mark"). The record's comment says so.
• They name the mechanism: mistune 3.4 parses a label before it knows the text is a label, so plugins never see the link or image state set.
• Every word but 3.4 is a non-version word.
• I checked each word with grading v2's mentions against:
  • the whole captured base output (244 lines);
  • the whole laid-over tests/test_mentions.py at the fix sha;
  • the symptom words (mention, link text, alt text), in both directions;
  • the names of the functions the fix edits (_parse_jira_mention, _inline_to_adf, _create_media_node, _contains_jira_mention_literal, _contains_image) and of the failing tests;
  • the base src/marklassian/converter.py (git show 3ebadd3:src/marklassian/converter.py from the repo cache).
• No hits. After the review fix, the remaining words were re-checked against a fresh containerized base run (6 failed, 20 passed), the laid-over test file and the base converter.py: still no hits.
• As a control, in_link hits the base source, as expected. That is why it and in_image are not cause words. refactor was dropped as too generic, and bare link parser was narrowed to new link parser.
• Leak review: I read the whole laid-over file at the fix sha. It has no comments or docstrings about mistune, parsing order or versions. Its test names describe only the behaviour (..._link_context_remains_non_mention, ..._image_alt_preserves_complete_formatted_text). The fix's own converter comment states the cause, but the agent never sees the fix tree.
• New version word: 3.4. It is added to the expected set in test_the_cause_words_grader_v2_skips_are_exactly_the_versions.

Rejected candidates

This batch:

• Support pint 0.26 dalito/ucumvert#84 (pint 0.26, issue fix(keys): agent keys can no longer approve or reject entries (T56) #82):
  • The error names its own fix ("the alias i.U. is not a valid unit alias").
  • The laid-over tests/test_ucum_pint.py has a comment stating the cause ("pint >= 0.26.2 rejects "[" and "]" as unknown operators") and a test name stating the fix (..._reaches_pint_without_brackets_...).
  • Laying over only tests/test_cli.py leaves a repr change that touches tests only.
• fix(ci): support SQLAlchemy 2.1 URL decoding and ADBC 1.13 nullability drls-io/sqlalchemy-adbc#21 (SQLAlchemy 2.1 URL.database quoting): the change is SQLAlchemy ticket 11234 in the 2.1.0b1 section of changelog_21.rst, released before the cutoff.
• Fix Q&A Excel compatibility with Markdown 3.11 superdd-coder/SinkDuce#3 (Markdown 3.11 removed the not_strong/em_strong2 patterns): the KeyError names the missing pattern. The laid-over test's name and its deregister("not_strong") monkeypatch show the fix. The repo is 61 MB.
• Fix automatic capture for newer aiohttp and urllib3 clients leadpoet/leadpoet#619 (aiohttp 3.14 StreamReader attributes): 122 MB repo, a Bittensor deployment with a large dependency set, and the fix mixes aiohttp and urllib3 2.8 changes.
• fix(ha): anyio task group in HA websocket proxy (fixes red CI) floco/myhome#153 (anyio 4.16 CancelScope propagates native Task.cancel()): the fix changes no test.
• Accept a cancelled close future in the finalizer cleanup Portocolom-Studio/potocolom#802 (same anyio change): the fix changes only a test.
• fix(LH-134871): support urllib3 2.8 in Python SDK cisco-lockhart/cdo-public-api-docs#127 (urllib3 2.8): the fix is only a pin, and the repo has no license.
• test: adapt to shapely 2.2.0 CS-SI/eodag#2378, test(python/sedonadb-geopandas): don't compare negative dwithin distances against geopandas apache/sedona-db#1422 (shapely 2.2): titles say tests only; not pursued.
• fix: declare click as a direct dependency pragmatiks/pragma-cli#160 (typer 0.28 dropped click): a dependency declaration only.
• Starlette 1.10 (TestClient on httpx2): the downstream PRs found add a dependency, not code.
• No usable downstream fixes turned up for more-itertools 11.2, trio 0.35, sphinx 9.2 (docs builds and Python-version bumps), pluggy 1.7 (ScanCode pins, plus the rejected decisionsmith and discord/access), cattrs 26.2 or pyarrow 26.

From the brief, not re-checked: harlequin#1208, constellations#719, datagrunt#375, fakesnow#457/#459, marimo#11525, uk_address_matcher#636, citibike-warehouse#2, fraud-ring-detection#3, DuckDB_ML#1, calcofi4py#2, ftw-global-data-catalog#32, datarecord#103, credit-matrix#160, light-labs#98, lawless-waf#27, decisionsmith#21, discord/access#709, dace#2634, covjson-msgspec#276, vcrpy#1094, sqlmodel-export#2, platefold#33, tugboat#146, requests-http-signature#45, skosprovider_sqlalchemy#129.

The candidates in PR #164, #165 and #167 were not re-checked either.

Why

Ticket: MIT-618 (fourth batch), the GitHub-task eval. These are real post-cutoff tasks for the arms run.

How it was tested

• Containerized runs of the new task for the base and the fix on mistune 3.4.0, and for the base on 3.3.4, as described above.
• server/tests/test_eval_grader.py and server/tests/test_eval_repos.py pass.
• make check passes after merging origin/feature/eval-setup (already up to date).

Learned

• A uv pip install <pin> after -e . installs a version above the project's own cap (mistune<3.4), so a task can pin a release the base refuses.
• A fix PR's own claim that its new test "also passes on the old code" means old code on the old version. Check the new test at the base on the new version too: marklassian's fails there.
• Mining by quoted release strings ("anyio 4.16", "urllib3 2.8", "sphinx 9.2") mostly finds lockfile refreshes, security bumps and docs-build changes. The few code fixes found were either missing a test change or touched only tests. Issues that name a mechanism would be a better place to search.
• A changelog's ticket number dates a change: SQLAlchemy ticket 11234 sits in the 2.1.0b1 section, so it shipped before the cutoff even though 2.1.0 came out in September.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y