From 2bdfe9ac774d9b30fa322599010220595a2bc293 Mon Sep 17 00:00:00 2001 From: Yuma Takino <36617106+tappe9@users.noreply.github.com> Date: Mon, 28 Sep 2026 00:12:07 +0900 Subject: [PATCH 1/2] docs: align implementation guides with registry and published results --- docs/AXUM.md | 17 +-- docs/FLASK.md | 7 +- docs/IMPLEMENTATIONS.md | 77 ++++++++++---- docs/JAVA-SPRING-BOOT.md | 14 +-- docs/NODE-EXPRESS.md | 6 +- tests/test_benchmark_implementation_docs.py | 109 ++++++++++++++++++++ 6 files changed, 196 insertions(+), 34 deletions(-) create mode 100644 tests/test_benchmark_implementation_docs.py diff --git a/docs/AXUM.md b/docs/AXUM.md index eab72a9..4a606a5 100644 --- a/docs/AXUM.md +++ b/docs/AXUM.md @@ -2,17 +2,17 @@ ## Status and comparison boundary -`rust-axum` is an independent implementation under `apps/rust-axum/`, registered -for normal CI and the unchanged shared API contract. It is **not** a member of the -active `four-stack-v1` cohort. Adding it does not change `results/latest.json`, -historical results, numeric README blocks, or the existing Actix application. -Activating a complete expanded cohort requires a separate reviewed change; see -[implementation registration](IMPLEMENTATIONS.md). +`rust-axum` is an independent implementation under `apps/rust-axum/`, with the +unchanged shared API contract. See the shared +[registration, CI and cohort status](IMPLEMENTATIONS.md#registered-and-measured-implementations) +for its current comparison and publication boundaries. Registration, cohort +activation and verified publication are separate milestones; published reports +retain their original measured source and versions. Axum uses its own router, Hyper HTTP server, Tower service integration and Tokio executor. Actix uses its Actix server/worker model. Their complete runtime and HTTP stacks differ even with the same compiler, SQLx, serializer and resource -limits. A future comparison is not a language-only or framework-only ranking. +limits. A comparison is not a language-only or framework-only ranking. The short diagnostic below cannot establish an official performance ordering. ## Pinned build and runtime @@ -127,7 +127,8 @@ the printed project name and remove only that owned project manually. ## Normal CI integration -The registry generates the normal six-implementation CI matrix. Axum's job runs +The normal matrix follows the shared [CI coverage](IMPLEMENTATIONS.md#ci-coverage), +not a fixed implementation count or an official-cohort-only list. Axum's job runs its acceptance, focused shared contract and then this diagnostic as required steps, with read-only repository permissions. The existing `required` aggregate rejects any failed, cancelled or skipped matrix job. The Axum job additionally diff --git a/docs/FLASK.md b/docs/FLASK.md index 86435f6..35f952f 100644 --- a/docs/FLASK.md +++ b/docs/FLASK.md @@ -50,4 +50,9 @@ The gates cover unit behavior, dependency/hash locks, real PostgreSQL and Docker ## Publication boundary -Python / Flask is registered and CI-covered, but Issue #32 does not activate a new official benchmark cohort. `active_cohort` remains the frozen `four-stack-v1`; expanded-cohort activation and new official measurements are separate work. +See the shared [registration, CI and cohort status](IMPLEMENTATIONS.md#registered-and-measured-implementations) +for Flask's place in the comparison. The production profile above describes +current main, not the server used in older measurements. The +[published-result boundary](IMPLEMENTATIONS.md#published-results-and-runtime-changes) +distinguishes the recorded Waitress result from the current Gunicorn runtime; +changing the runtime does not update or relabel published measurements. diff --git a/docs/IMPLEMENTATIONS.md b/docs/IMPLEMENTATIONS.md index a61b408..6150835 100644 --- a/docs/IMPLEMENTATIONS.md +++ b/docs/IMPLEMENTATIONS.md @@ -2,8 +2,8 @@ ## Sources of truth -`benchmark/implementations.json` is the small, versioned registry for implemented -stacks. An entry contains the stable implementation ID, language, framework, +[`benchmark/implementations.json`](../benchmark/implementations.json) is the small, +versioned registry for implemented stacks. An entry contains the stable implementation ID, language, framework, display name, source/build directory, required version fields, service acceptance test, and optional focused acceptance-failure test. The Compose service name is the implementation ID; its build context must equal the registered source path. @@ -36,40 +36,81 @@ plugin dependency is introduced. ## Registered and measured implementations -The nine registered implementations, in deterministic registry order, are: +The registered implementations, in deterministic registry order, are: Go / Gin, Go / Echo, Rust / Actix Web, Rust / Axum, Node.js / Fastify, -Node.js / Express, Python / FastAPI, Python / Flask, and Java / Spring Boot. Registration enables -source checks, real-container acceptance and unchanged shared contracts; it does -not by itself activate a benchmark cohort or create measured results. +Node.js / Express, Python / FastAPI, Python / Flask, and Java / Spring Boot. +Registration means a stack has source and validation entries in the registry; +it does not by itself activate a benchmark cohort or create measured results. +Cohort membership selects the complete group for official measurement. Actual +publication is separate evidence, tied to the measured source rather than to +current registration or CI success. | Cohort | Ordered members | Current role | | --- | --- | --- | | `four-stack-v1` | `go-gin`, `rust-actix`, `node-fastify`, `python-fastapi` | Frozen historical cohort; existing reports remain readable | | `eight-stack-v1` | `go-gin`, `go-echo`, `rust-actix`, `rust-axum`, `node-fastify`, `node-express`, `python-fastapi`, `python-flask` | Active official cohort; complete verified runs are eligible for publication | -`registered_pinned_versions()` in `benchmark/environment.py` extracts all nine -stacks, while `pinned_versions()` includes only the active cohort's metadata. -Axum additionally records its explicit Tokio version. The +`registered_pinned_versions()` in `benchmark/environment.py` extracts all +registered stacks, while `pinned_versions()` includes only the active cohort's +metadata. Axum additionally records its explicit Tokio version. The [Axum-only diagnostic](AXUM.md) consumes registered Axum metadata separately and does not introduce partial official reports. +### CI coverage + +The normal CI matrix covers all registered implementations, not just members of +the active official cohort. `python -m benchmark.ci matrix` derives each entry +from the registry. Each implementation job runs its acceptance/failure gates and +focused shared API contract; Axum also runs its required non-publishing load +diagnostic. The separate smoke job selects the active cohort and cannot publish. +The `required` aggregate rejects failed, cancelled or skipped dependencies; CI +success proves those checks passed, not official measured performance. See the +[CI workflow](../.github/workflows/ci.yml) for the executable gates. + +### Published results and runtime changes + +The [verified report published on 2026-09-15](../results/history/2026-09-15T04-14-09Z-34925168324-1.json) +contains Axum and Flask, with Flask measured using Waitress 3.0.2. It is still the +[latest published result](../results/latest.json) at this documentation update. +The report's `metadata.source_commit`, runtime versions and conditions describe +that measurement, not whichever code is now on main. + +Current Flask uses Gunicorn with one `gthread` request worker and one request +thread, plus its arbiter process; see the [Flask runtime guide](FLASK.md). +The recorded Waitress result is not a Gunicorn performance measurement. +The runtime change in PR #77 did not rerun the official benchmark. Never replace +recorded server/version metadata with current pins, relabel old measurements or +infer a performance change from CI. A new official result requires a separately +authorized complete run and validated publication. README and Pages continue to +display the selected report's measured configuration until then. + ### Java registration boundary (#67) -`java-spring-boot` is registered with a Java-only host toolchain, real-container -acceptance and the unchanged shared contract. Its static version extractor reads -the committed JDK build, Gradle Wrapper, strict lockfile and SHA-256 verification +`java-spring-boot` is registered and CI-covered with a Java-only host toolchain, +real-container acceptance and the unchanged shared contract. Its static version +extractor reads the committed JDK build, Gradle Wrapper, strict lockfile and SHA-256 verification metadata without executing Java or Gradle. Other implementation jobs do not need a host JDK. See [Java / Spring Boot](JAVA-SPRING-BOOT.md). -Neither cohort changes membership or order. Official metadata still contains -only the eight active implementations; Java is not filled into old reports with -zeroes or missing values. A future Java-inclusive cohort and any official run -need a separate decision, including an assessment of JVM warm-up under the common -profile. Registration and passing tests are not measured performance evidence. +Java is not a member of either official cohort in the table above, and there is +no official Java performance result. Official metadata includes only the selected +cohort; Java is not filled into existing reports with zeroes or missing values. +A future Java-inclusive cohort and any official run need a separate decision, +including an assessment of JVM warm-up under the common profile. Registration +and passing tests are not measured performance evidence. + +### Historical introduction and releases + +The original Axum, Express and Flask registration work preceded the expanded +cohort's activation. The `four-stack-v1` boundary described at introduction was +historical rollout context, not a current exclusion from official comparison. +The v0.1.0 release remains the original four-stack snapshot; later registration, +activation and publication on main do not rewrite that release or its results. +See [release snapshots](RELEASING.md) and the frozen historical cohort below. ### Eight-stack rollout boundary (#50) -The approved activation sets `active_cohort: eight-stack-v1`. Both readers accept +The approved activation in Issue #50 set `active_cohort: eight-stack-v1`. Both readers accept complete four/eight-stack reports, but no synthetic fixture is published and the active pointer is not evidence of a real eight-stack measurement. README and Pages identify the selected report's cohort, definition and API diff --git a/docs/JAVA-SPRING-BOOT.md b/docs/JAVA-SPRING-BOOT.md index cd755d3..c50a69f 100644 --- a/docs/JAVA-SPRING-BOOT.md +++ b/docs/JAVA-SPRING-BOOT.md @@ -2,11 +2,13 @@ ## Registration is not official measurement -`java-spring-boot` is the ninth registered implementation. CI builds it and checks -it against the same API contract as the other implementations. It is **not** a -member of `eight-stack-v1` or frozen `four-stack-v1`. Neither historical data nor -the current result receives synthetic Java values. There is no official Java -performance result from adding or testing this implementation. +`java-spring-boot` is registered and CI-covered under the same API contract as the +other implementations. See the shared +[registration, CI and cohort status](IMPLEMENTATIONS.md#registered-and-measured-implementations) +and [Java registration boundary](IMPLEMENTATIONS.md#java-registration-boundary-67). +Java is not in an official cohort and has no official performance result. +Neither historical data nor the current result receives synthetic Java values; +adding or testing an implementation is not official measurement. ## Fixed build inputs @@ -106,7 +108,7 @@ The CI matrix derives `java` from the registry language. Only its Java entry installs the pinned JDK. Other language acceptance jobs guard Java executables alongside their other unrelated host toolchains. Shared version extraction reads committed files without invoking Java or Gradle. The existing fail-closed -`required` aggregate must include the new implementation; no gate is optional. +`required` aggregate includes Java; no gate is optional. ## Updating dependencies or measuring Java diff --git a/docs/NODE-EXPRESS.md b/docs/NODE-EXPRESS.md index 432e60c..2ca6d95 100644 --- a/docs/NODE-EXPRESS.md +++ b/docs/NODE-EXPRESS.md @@ -6,4 +6,8 @@ This is a complete-stack comparison, not an isolated claim about Express routing The implementation provides `/health`, `/json`, `/db/:id`, and `/cpu`, preserves signed PostgreSQL BIGINT values as JSON numbers, uses direct recursive Fibonacci(30), runs as the non-root `node` user, and closes the HTTP server before the database pool during SIGINT/SIGTERM shutdown. -Express is registered and exercised by normal CI, including unit tests, real-container acceptance and the shared API contract. The official benchmark remains `four-stack-v1`; adding this implementation does not publish a partial expanded-cohort result. +Express is exercised by normal CI, including unit tests, real-container +acceptance and the shared API contract. See the shared +[registration, CI and cohort status](IMPLEMENTATIONS.md#registered-and-measured-implementations) +for its comparison and publication boundaries. Registration and passing tests +do not themselves publish a result or change historical measurements. diff --git a/tests/test_benchmark_implementation_docs.py b/tests/test_benchmark_implementation_docs.py new file mode 100644 index 0000000..cbbcd39 --- /dev/null +++ b/tests/test_benchmark_implementation_docs.py @@ -0,0 +1,109 @@ +"""Keep implementation status in one registry-backed guide, not rollout snapshots.""" + +import json +import re +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +DOCS = ROOT / "docs" +STATUS = "IMPLEMENTATIONS.md#registered-and-measured-implementations" +GUIDE_SECTIONS = { + "AXUM.md": "Status and comparison boundary", + "FLASK.md": "Publication boundary", + "JAVA-SPRING-BOOT.md": "Registration is not official measurement", + "NODE-EXPRESS.md": None, +} + + +def section(text: str, heading: str) -> str: + """Read one heading and its prose, without matching a whole paragraph.""" + match = re.search(rf"(?m)^#{{2,3}} {re.escape(heading)}\n", text) + if match is None: + raise AssertionError(f"Missing documentation section: {heading}") + return re.split(r"(?m)^#{2,3} ", text[match.end() :], maxsplit=1)[0] + + +class ImplementationDocumentationTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.registry = json.loads((ROOT / "benchmark/implementations.json").read_text()) + cls.canonical = (DOCS / "IMPLEMENTATIONS.md").read_text() + + def assert_canonical_link(self, text, target): + links = re.findall(r"\[[^\]]+\]\(([^)]+)\)", text) + self.assertIn(target, links) + path, anchor = target.split("#", 1) + destination = DOCS / path + self.assertTrue(destination.is_file(), target) + headings = re.findall(r"(?m)^#{1,6} (.+)$", destination.read_text()) + anchors = { + re.sub(r"[^\w\- ]", "", heading.lower()).replace(" ", "-") for heading in headings + } + self.assertIn(anchor, anchors, target) + + def test_status_sections_delegate_cohort_claims_to_canonical_guide(self): + for filename, heading in GUIDE_SECTIONS.items(): + with self.subTest(guide=filename): + text = (DOCS / filename).read_text() + current = section(text, heading) if heading else text + self.assert_canonical_link(current, STATUS) + # Cohort identities belong in the shared guide, not per-guide snapshots. + for cohort in self.registry["cohorts"]: + self.assertNotIn(f"`{cohort}`", current) + + def test_canonical_cohort_table_matches_registry_membership_and_active_pointer(self): + rows = {} + table = section(self.canonical, "Registered and measured implementations") + for implementation in self.registry["implementations"]: + self.assertIn(implementation["display_name"], table) + for line in table.splitlines(): + cells = [cell.strip() for cell in line.split("|")] + if len(cells) == 5 and cells[1].startswith("`"): + cohort = cells[1].strip("`") + self.assertNotIn(cohort, rows) + rows[cohort] = (re.findall(r"`([^`]+)`", cells[2]), cells[3]) + self.assertEqual(set(rows), set(self.registry["cohorts"])) + for cohort, expected in self.registry["cohorts"].items(): + with self.subTest(cohort=cohort): + members, role = rows[cohort] + self.assertEqual(members, expected["members"]) + self.assertEqual( + "Active official cohort" in role, cohort == self.registry["active_cohort"] + ) + + def test_ci_scope_is_registry_wide_and_axum_links_to_it(self): + coverage = section(self.canonical, "CI coverage") + self.assertRegex(" ".join(coverage.split()), r"all registered implementations") + self.assertIn("benchmark.ci matrix", coverage) + self.assertIn("`required`", coverage) + axum = section((DOCS / "AXUM.md").read_text(), "Normal CI integration") + self.assert_canonical_link(axum, "IMPLEMENTATIONS.md#ci-coverage") + self.assertNotRegex(axum, r"\b\w+-implementation CI matrix\b") + + def test_flask_runtime_change_is_distinct_from_dated_measurement(self): + boundary = section(self.canonical, "Published results and runtime changes") + paragraphs = [" ".join(p.split()) for p in boundary.split("\n\n")] + published = next((p for p in paragraphs if "2026-09-15" in p), "") + self.assertRegex(published, r"Waitress(?:\s|`)+3\.0\.2") + self.assertIn("source_commit", boundary) + self.assertIn("Gunicorn", boundary) + self.assertRegex(boundary, r"(?i)not.*Gunicorn.*measurement") + self.assertIn("results/history/2026-09-15T04-14-09Z-34925168324-1.json", boundary) + flask = section((DOCS / "FLASK.md").read_text(), "Publication boundary") + self.assert_canonical_link(flask, "IMPLEMENTATIONS.md#published-results-and-runtime-changes") + + def test_java_boundary_does_not_claim_official_results(self): + java = " ".join(section(self.canonical, "Java registration boundary (#67)").split()) + self.assertIn("`java-spring-boot`", java) + self.assertRegex(java, r"(?i)registered.*\bCI\b") + self.assertRegex(java, r"(?i)not.*member.*official cohort") + self.assertRegex(java, r"(?i)no official Java.*result") + guide = section( + (DOCS / "JAVA-SPRING-BOOT.md").read_text(), "Registration is not official measurement" + ) + self.assert_canonical_link(guide, "IMPLEMENTATIONS.md#java-registration-boundary-67") + + +if __name__ == "__main__": + unittest.main() From 5eb7084dd4fb01324c4f3312c188991a0edaa6ce Mon Sep 17 00:00:00 2001 From: Yuma Takino <36617106+tappe9@users.noreply.github.com> Date: Mon, 28 Sep 2026 00:20:48 +0900 Subject: [PATCH 2/2] style: apply pinned Ruff formatting to documentation tests --- tests/test_benchmark_implementation_docs.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/test_benchmark_implementation_docs.py b/tests/test_benchmark_implementation_docs.py index cbbcd39..4ea6e1d 100644 --- a/tests/test_benchmark_implementation_docs.py +++ b/tests/test_benchmark_implementation_docs.py @@ -91,7 +91,9 @@ def test_flask_runtime_change_is_distinct_from_dated_measurement(self): self.assertRegex(boundary, r"(?i)not.*Gunicorn.*measurement") self.assertIn("results/history/2026-09-15T04-14-09Z-34925168324-1.json", boundary) flask = section((DOCS / "FLASK.md").read_text(), "Publication boundary") - self.assert_canonical_link(flask, "IMPLEMENTATIONS.md#published-results-and-runtime-changes") + self.assert_canonical_link( + flask, "IMPLEMENTATIONS.md#published-results-and-runtime-changes" + ) def test_java_boundary_does_not_claim_official_results(self): java = " ".join(section(self.canonical, "Java registration boundary (#67)").split())