Sync main into the Kitesurf branch - #18
Merged
Merged
Conversation
From a file inside docs/, ../../ lands on /blob/ rather than the repository root, so the link to the Kitesurf lane's index resolved to /blob/tree/kitesurf-eval/... and 404ed. The same form is correct in README.md because that file sits at the root, one level higher.
* Name the evidence release by when its runs were collected The release holds full run products, so the tag now reads evidence-20260812 rather than a version number. That names the drop instead of the code, which is the useful identity for someone looking for a particular run's logs. It also needs saying out loud that the two dates differ: the runs were collected on 2026-08-12 and 08-13, while the tree this tag points at is later, so the tag is not a claim about the state of the code. * Break the release paragraph where the Chinese page breaks it
The README story is rewritten in both languages: a tagline badge in deep blue, a coverage strip naming the task count and the automation tools, a 13-driver table with ecosystems, control paths and pinned versions, and a collapsed five-engine glance under the Kitesurf note. Numbers now state what they count: the scenario expansion explains why 116 specs yield 1,233 tasks, the L2 line separates the 188 tasks from the l2.web_platform subset and derives the 192 scoring units, and the intro no longer double-counts WebDriver next to the 13 drivers. Across the docs, caliber wording becomes task subsets, chrome_gate_fail is no longer mentioned since the published configuration cannot produce it, and the remaining em dashes in the English docs are gone.
…ery table (#8) The collapsed five-engine block now sits directly under the four-engine table, before the run parameters, so the wider comparison is one click away from the leaderboard. The coverage strip drops the language-ecosystems item; the tool names already carry the breadth and the driver table spells out the ecosystems. Every table is wrapped in a centered block, and the English Kitesurf note loses its evidence-class clause to match the trimmed Chinese one.
Two commands lifted verbatim from the shortest path in RUNNING, a one-line prerequisite list, the reminder that a single-engine run never scores formally, and a pointer that handing RUNNING to a coding agent is the fastest way to get the bench running.
The package version arrived at 0.5.0 in the first commit, with no 0.1 through 0.4 behind it and no record of what the 5 stood for. Sitting next to a corpus labelled v0_4, it read as the next step in that sequence, which it never was: one number counts corpus generations, the other counts releases of this code. 0.1.0 says what is true, that this is the first public cut of the harness, and no longer invites the comparison. Both manifests and the two root entries in the lockfile move together; dependency versions and the license are untouched.
The repo carried three version identifiers in two styles, with nothing stating how they relate: `bench_version` (`2026.08.17-v0_4.2`), `site.site_version` (`2026.07.10-fixtures`, read by no code at all), and the harness `0.1.0`. The `v0_4.x` part is lineage from a private repository and carries no meaning here. Two axes, one style. `bench_version` becomes plain semver and its shape is now enforced by `validate`, so the number answers the only question it exists for: whether two runs are comparable. MAJOR is task membership, graders, fixtures or capability assignments; MINOR is additive; PATCH is descriptions and other non-executable metadata. The published runs sit at `0.4.1` and HEAD at `0.4.2`, which under these rules is a PATCH apart — hence the numbering rather than a fresh `1.0.0`, which would have read as an incompatible dataset. The fixture tree belongs to the dataset axis, so `site_version` is dropped rather than kept in sync; a run already records `runner.fixtures.tree_sha256`, which is checkable where a hand-maintained label is not. The harness version moves to `runner/version.py` as the single source `pyproject.toml` reads, and runs now record it as `harness_version`. Reports keep regenerating byte-identically from the published evidence: runs predating this scheme carry no `harness_version`, and the report header only gains the field when one is present.
The branch had been sitting on its fork point since it was created, so every main-side change since then was missing here: the unified versioning scheme, the Apache-2.0 relicense, the harness package version, and the README and documentation revisions including the Chinese refinements. Shared documents take main's wording; branch-local framing is kept where the two differ on purpose. This branch's README is a delta document rather than a second front page, so main's Quick Start, Benchmark Composition and Repository Layout sections are not pulled in, and its own documentation table — a superset of main's, adding the deployment and evidence entries — stays. `tools/report_five_engine.py` gets the same treatment its four-engine counterpart got: the header names the harness version only when the run recorded one, so the published report still regenerates byte-identically from frozen evidence. `config/kitesurf_dynamic_fixture.json` pins the runner source digest, which moved because the versioning change edited `runner/run.py`. Three hashes are updated: the implementation digest, the response digest of the probe that fetches the contract, and the contract digest itself. Nothing else in the contract changed, and its verifier passes against a local FixtureServer.
test_consolidated_branch_descends_from_recorded_main asks git whether this branch descends from the main commit recorded in config/kitesurf_experiments.json. actions/checkout defaults to a shallow clone, which does not contain that commit, so git exits 128 and the assertion fails on the error rather than on the answer. Every Kitesurf pull request has failed this way, including the three that were merged regardless. Fetching the full history costs little on a repository this size and keeps the check doing what it was written to do. Main does not run this test, so the same line there is harmless but not required.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch had been sitting on its fork point since it was created, so everything main did afterwards was missing here. This merges main in.
What comes across
0.4.2, harness version inrunner/version.py,site_versiongone,validateenforcing semver.0.1.0harness package version.How the shared documents were resolved
Shared documents take main's wording. Branch-local framing is kept where the two differ on purpose — self-referential phrasing ("you are on
kitesurf-eval"), links to this branch's own report, and the in-page anchor to the Kitesurf lane section rather than a cross-branch link.This branch's README is a delta document, not a second front page, so main's Quick Start, Benchmark Composition and Repository Layout sections are deliberately not pulled in, and its own documentation table — a superset of main's, adding the deployment and evidence entries — stays. The Chinese README picks up main's translated headings and the refined resource sentence.
Two branch-only follow-ons
tools/report_five_engine.pygets the same treatment as its four-engine counterpart: the header names the harness version only when the run recorded one. The published five-engine report's generated sections still come out byte-identical from the frozen evidence.config/kitesurf_dynamic_fixture.jsonpins the runner source digest, which moved because the versioning change editedrunner/run.py. Three hashes are updated — the implementation digest, the response digest of the probe that fetches the contract, and the contract digest itself. Nothing else in the contract changed. Worth knowing going forward: any change torunner/on main invalidates this contract and requires the same regeneration here.A pre-existing CI failure, fixed here
test_consolidated_branch_descends_from_recorded_mainasks git whether this branch descends from the main commit recorded inconfig/kitesurf_experiments.json.actions/checkoutdefaults to a shallow clone, which does not contain that commit, so git exits 128 and the assertion fails on the error rather than on the answer. Every Kitesurf pull request has failed this way, including #7, #9 and #12, which were merged regardless. This branch's CI now fetches the full history, which costs little here and lets the check do what it was written to do. Main does not run this test, so the same line there is harmless but not required.Checks
validate,scenarios --check,pytest: 717 passed, 9 skipped