Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,15 @@ jobs:
- name: Install pinned Rust toolchain
if: matrix.toolchain == 'rust'
run: rustup toolchain install 1.98.1 --profile minimal --component rustfmt,clippy
- name: Install pinned Java toolchain
if: matrix.toolchain == 'java'
run: |
curl --fail --silent --show-error --location --max-time 180 'https://github.com/adoptium/temurin25-binaries/releases/download/jdk-25.0.4.1%2B1/OpenJDK25U-jdk_x64_linux_hotspot_25.0.4.1_1.tar.gz' -o "$RUNNER_TEMP/java.tar.gz"
echo "dbb698396d478e7fa2b1e50f4103324b2a99b90569ee27c33f2261f9215cf41e $RUNNER_TEMP/java.tar.gz" | sha256sum -c -
mkdir -p "$RUNNER_TEMP/java"
tar -xzf "$RUNNER_TEMP/java.tar.gz" --strip-components=1 -C "$RUNNER_TEMP/java"
echo "JAVA_HOME=$RUNNER_TEMP/java" >> "$GITHUB_ENV"
echo "$RUNNER_TEMP/java/bin" >> "$GITHUB_PATH"
- name: Run implementation acceptance and failure gates
run: python -m benchmark.ci run-isolated --implementation "$IMPLEMENTATION_ID" -- make "test-$IMPLEMENTATION_ID"
- name: Run focused shared contract against implementation
Expand Down
31 changes: 27 additions & 4 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The architecture is intentionally small.
1. A new reader should understand the repository quickly.
2. Every backend should implement the same API contract.
3. A local benchmark should run with one command.
4. GitHub Actions should measure every backend in the same job.
4. GitHub Actions should measure every active cohort member sequentially in the same job.
5. README and GitHub Pages should read the same result file.

## Non-goals for v0.1
Expand All @@ -36,6 +36,8 @@ flowchart LR
R --> A2[Rust / Actix Web and Axum]
R --> A3[Node.js / Fastify and Express]
R --> A4[Python / FastAPI and Flask]
M -. acceptance and contracts only .-> A5[Java / Spring Boot]
A5 --> P
A1 --> P[(PostgreSQL)]
A2 --> P
A3 --> P
Expand Down Expand Up @@ -100,6 +102,26 @@ The lifespan creates a pool with `min_size=1` and `max_size=10`, executes `SELEC

Uvicorn is the direct container process, with `--workers 1 --loop asyncio --http h11`, no access log, and a five-second graceful HTTP shutdown timeout. The lifespan closes the pool with a five-second bound and terminates remaining connections only if graceful pool closure fails or is cancelled. The production image installs only the SHA256-verified runtime lock and excludes tests and development tools. Both Docker stages use `python:3.14.7-slim-bookworm` pinned to index digest `sha256:9ab8d9c8514b44f90cf0029dd42fdd7e9e211e639c8b995304cc04568dee900f`. Compose uses non-root user `10001:10001`, 1 CPU / 512 MB, healthy PostgreSQL ordering, dropped capabilities, no privilege escalation, loopback-only port `8080`, and no restarts.

#### Java / Spring Boot

`apps/java-spring-boot/` uses a single Temurin 25.0.4.1+1 JVM with Spring Boot
4.1.1, Spring MVC/Tomcat, ordinary record serialization and JDBC/HikariCP.
`DatabaseSettings` validates the shared five `DATABASE_*` values without putting
credentials into the JDBC URL. `DatabaseConfiguration` bounds the pool at ten
connections and validates `SELECT 1` before startup completes. Repository reads
use a prepared statement and try-with-resources; signed BIGINT IDs remain numeric
JSON, missing rows return 404, and SQL failures return a sanitized 500. Each CPU
request calculates Fibonacci(30) by direct recursion.

The production image is non-root, inherits the common resource/isolation limits,
contains only runtime libraries and shuts down HTTP and the pool on SIGTERM.
Architecture-selected BuildKit stages checksum-verify the exact JDK/JRE archives;
both OS base images are digest pinned. Gradle Wrapper and dependency verification
are strict. The small standalone health probe does not start Spring or connect
to the database. No ORM, WebFlux, native image, virtual threads or additional
server processes are enabled. Java is CI-covered but not an official cohort
member; see [the Java guide](docs/JAVA-SPRING-BOOT.md).

### PostgreSQL

One PostgreSQL container is shared by all implementations. It runs as the `postgres` service on the project-scoped `benchmark` network without a published host port. API services connect on internal port `5432` using the common `DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_NAME`, `DATABASE_USER`, and `DATABASE_PASSWORD` settings defined in the Compose extension field.
Expand All @@ -115,7 +137,7 @@ required version fields and acceptance entry points. Python reads it directly;
`benchmark/registry.py` generates and checks the Make and JavaScript projections.
CI invokes registry-backed sequential acceptance targets and verifies Compose
build-context agreement. Version extraction remains language-specific.
`registered_pinned_versions()` covers all eight registered stacks; `pinned_versions()`
`registered_pinned_versions()` covers all nine registered stacks; `pinned_versions()`
filters that metadata to the active official cohort. Diagnostic metadata records
only the selected Axum stack, without expanding official reports.

Expand Down Expand Up @@ -189,7 +211,8 @@ simple-api-benchmark/
│ ├── node-fastify/
│ ├── node-express/
│ ├── python-fastapi/
│ └── python-flask/
│ ├── python-flask/
│ └── java-spring-boot/
├── benchmark/
│ ├── config.json
│ ├── definition.py
Expand Down Expand Up @@ -244,7 +267,7 @@ Cleanup must run even when a build, contract test, or benchmark fails.

### Pull requests

`ci.yml` builds and checks all eight registered applications, runs the shared contract and focused tests, validates workflow syntax/security and generated README sections, validates the static Pages contracts, and executes a short non-publishing smoke benchmark. Repository permissions are read-only and checkout credentials are not persisted. The Axum matrix entry additionally runs its non-publishing diagnostic and uploads diagnostic evidence; failure blocks the existing required aggregate. The active-cohort smoke covers `eight-stack-v1` sequentially. The component diagram groups applications by language; the runner still starts and measures only one implementation at a time.
`ci.yml` builds and checks all nine registered applications, runs the shared contract and focused tests, validates workflow syntax/security and generated README sections, validates the static Pages contracts, and executes a short non-publishing smoke benchmark. Repository permissions are read-only and checkout credentials are not persisted. The Axum matrix entry additionally runs its non-publishing diagnostic and uploads diagnostic evidence; failure blocks the existing required aggregate. The active-cohort smoke covers `eight-stack-v1` sequentially. The component diagram groups applications by language; the runner still starts and measures only one implementation at a time.

<a id="scheduled-and-manual-benchmarks"></a>

Expand Down
27 changes: 24 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,19 +136,40 @@ The exact Python patch version is also recorded in `.python-version`. Tests use

When updating the Python baseline, review official releases and wheel availability for CPython 3.14, change the `.in` files and exact Python version intentionally, resolve the full runtime and development graphs in a clean environment, and record published wheel SHA256 hashes in both lock files. Review transitive changes, verify the official Docker index digest, and update the matching acceptance expectations and documentation. Never install from an unlocked `.in` file for validation or production. Rerun all DB/API acceptance targets after shared Compose changes.

### Java / Spring Boot checks

Install the exact Temurin JDK recorded in `apps/java-spring-boot/.java-version`
(currently `25.0.4.1+1`), plus Python 3.10+, Make and Docker Compose v2 on a POSIX
host. A global Gradle installation is not needed. Run the native checks without
Docker, or the complete production acceptance target:

```bash
(cd apps/java-spring-boot && ./gradlew --no-daemon test installDist)
make test-java-spring-boot
make test-contract CONTRACT_IMPL=java-spring-boot
```

The committed Wrapper checksum, distribution checksum, strict dependency locks
and artifact verification metadata are mandatory. Validation must not regenerate
those files or download an alternative JDK silently. The production acceptance
checks live SQL/BIGINT responses, JSON errors, pool and container limits, finite
startup failure, SIGTERM shutdown and project-scoped cleanup. See the
[Java guide](docs/JAVA-SPRING-BOOT.md) for the dependency-update procedure.
Java registration does not expand the active official cohort or authorize a run.

### Shared contract checks

The common suite needs Python 3.10+ on a POSIX host, Docker Compose v2 with
`up --wait` support, and Make. It uses only the Python standard library; language
toolchains are not needed on the host for this target.

```bash
make test-contract # all eight registered implementations, sequentially
make test-contract # all nine registered implementations, sequentially
make test-contract CONTRACT_IMPL=go-gin # one implementation, same assertions
```

`CONTRACT_IMPL` also accepts `go-echo`, `rust-actix`, `rust-axum`, `node-fastify`,
`node-express`, `python-fastapi`, and `python-flask`.
`node-express`, `python-fastapi`, `python-flask`, and `java-spring-boot`.
The target first runs the suite's focused tests, then builds and starts the selected
production services. All expected statuses and JSON objects are read from the
paired HTTP/JSON examples in `docs/API-CONTRACT.md`; that document remains the
Expand Down Expand Up @@ -238,7 +259,7 @@ prerequisites plus that implementation's pinned language toolchain, not every
language used by the repository. CI derives the language setup from the registry
and guards unrelated host executables during acceptance, contracts and the Axum
diagnostic. Local `make test-<ID>` commands are unchanged; `make test` still needs
all toolchains and shared workflow/site dependencies. See the
all toolchains, including the exact Java JDK, and shared workflow/site dependencies. See the
[host-toolchain contract](docs/AUTOMATION.md#implementation-host-toolchains).

A pull request must pass the available checks for the area it changes.
Expand Down
26 changes: 22 additions & 4 deletions README.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@

**Go・Rust・Node.js・Pythonを、同じAPI・同じ制限・同じ負荷で比較します。**

Simple API Benchmarkには、同じエンドポイント、同じDockerリソース制限、同じ検証ルールを使う8つのAPI実装があります。普遍的な最速言語を決めることではなく、誰でも理解できて、自分でも再実行できる小さな比較を目指します。
Simple API Benchmarkには、同じエンドポイント、同じDockerリソース制限、同じ検証ルールを使う9つのAPI実装があります。普遍的な最速言語を決めることではなく、誰でも理解できて、自分でも再実行できる小さな比較を目指します。

> **現在の状態:** v0.1.0は初期の4構成を収録したリリースです。現在の`main`は8実装すべてがCI対象で、有効な`eight-stack-v1`の検証済み結果を公開しています。Pagesではダッシュボードと過去の結果を閲覧できます。
> **現在の状態:** v0.1.0は初期の4構成を収録したリリースです。現在の`main`は9実装がCI対象です。公式比較は引き続き`eight-stack-v1`の8実装で、Pagesに検証済み結果と履歴を公開しています。Java / Spring Bootは検証対象として登録済みですが、公式測定・公開結果には含まれません。

8構成の最初の完全な公式結果は、2026年9月15日に
[Issue #50 / PR #58](https://github.com/tappe9/simple-api-benchmark/pull/58)で公開しました。
Expand Down Expand Up @@ -244,10 +244,28 @@ make test-python-fastapi PYTHON=python3.14

acceptance targetにはPOSIX環境のPython 3.14.7、Docker Compose v2、Makeが必要です。一時virtual environmentへhash検証付きで開発用依存をinstallし、Ruff、focused pytest tests、実Dockerサービス、DB更新・異常系、資源制限、1 worker、起動失敗、SIGTERM終了、container・network削除を確認します。Dockerを使わないfocused testsは[Contributing](CONTRIBUTING.md)を参照してください。

8つのAPI実装と共通contract suiteを利用できます。
## Java / Spring Boot実装

`apps/java-spring-boot/`はTemurin 25.0.4.1+1、Spring Boot 4.1.1、
Spring MVC / Tomcat、JDBC / HikariCP、チェックサムを検証するGradle 9.7.1を使用します。
共通のAPI、単一プロセス、1 CPU・512 MiB、DB接続上限10を維持します。
**9番目の登録済み実装ですが、公式測定対象への追加ではありません。**
`eight-stack-v1`と公開済みの測定値は変更しません。

```bash
make test-java-spring-boot
make test-contract CONTRACT_IMPL=java-spring-boot
```

受入テストには、POSIXホスト上の固定JDK、Python 3.10以降、Make、Docker Compose v2が必要です。
コンテナのみを使う共通契約テストには、ホストのJDKは不要です。
固定依存関係、起動・終了の検証、公式測定と分ける理由は
[Java実装ガイド](docs/JAVA-SPRING-BOOT.md)を参照してください。

9実装すべてを、共通のAPI契約で検証できます。

```bash
make test-contract # 8実装を1つずつ順番に検証
make test-contract # 9実装を1つずつ順番に検証
make test-contract CONTRACT_IMPL=go-echo # 1実装を同じ契約で検証
```

Expand Down
26 changes: 22 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@

**Go vs Rust vs Node.js vs Python — same API, same limits, simple results.**

Simple API Benchmark contains eight API implementations with the same endpoints, Docker resource limits, and validation rules. The goal is not to declare a universal winner. The goal is to make a small, repeatable comparison that anyone can understand.
Simple API Benchmark contains nine API implementations with the same endpoints, Docker resource limits, and validation rules. The goal is not to declare a universal winner. The goal is to make a small, repeatable comparison that anyone can understand.

> **Project status:** v0.1.0 is released as the original four-stack snapshot. Current `main` includes all eight CI-covered implementations, the active `eight-stack-v1` cohort with verified published results, and a Pages dashboard with history navigation.
> **Project status:** v0.1.0 is released as the original four-stack snapshot. Current `main` includes nine CI-covered implementations. The active `eight-stack-v1` cohort has verified published results and a Pages dashboard with history navigation. Java / Spring Boot is registered for validation but is not part of the official cohort or published results.

The first complete eight-stack official result was published on September 15, 2026
under [Issue #50 / PR #58](https://github.com/tappe9/simple-api-benchmark/pull/58).
Expand Down Expand Up @@ -244,10 +244,28 @@ make test-python-fastapi PYTHON=python3.14

The complete acceptance target requires Python 3.14.7 on a POSIX host, Docker Compose v2, and Make. It installs the hash-locked development dependencies in a temporary virtual environment, runs Ruff and focused pytest tests, and verifies the real Docker service, DB errors and updates, resources, one worker, startup failure, SIGTERM shutdown, and container/network cleanup. See [Contributing](CONTRIBUTING.md) for focused tests without Docker.

All eight API implementations and the shared contract suite are available:
## Java / Spring Boot implementation

`apps/java-spring-boot/` uses Temurin 25.0.4.1+1, Spring Boot 4.1.1,
Spring MVC/Tomcat, JDBC/HikariCP, and checksum-verified Gradle 9.7.1.
It follows the same API, single-process, 1 CPU / 512 MiB and ten-connection
limits. It is the **ninth registered implementation**, not an additional official
result. `eight-stack-v1` and all published measurements remain unchanged.

```bash
make test-java-spring-boot
make test-contract CONTRACT_IMPL=java-spring-boot
```

The acceptance target needs the exact JDK, Python 3.10+, Make and Docker Compose
v2 on a POSIX host. The container-only contract target does not need a host JDK.
See the [Java implementation guide](docs/JAVA-SPRING-BOOT.md) for build integrity,
startup/shutdown behavior, supported build paths and the unmeasured boundary.

All nine API implementations and the shared contract suite are available:

```bash
make test-contract # all eight APIs, one at a time
make test-contract # all nine registered APIs, one at a time
make test-contract CONTRACT_IMPL=go-echo # one API with the same contract
```

Expand Down
8 changes: 5 additions & 3 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ v0.1.0 release. A next release version remains an explicit maintainer decision.
| Milestone | Status and evidence |
| --- | --- |
| Eight API implementations | Complete: Gin, Echo, Actix Web, Axum, Fastify, Express, FastAPI and Flask are registered and CI-covered. Axum retains its required non-publishing diagnostic. |
| Java / Spring Boot registration | Ninth implementation with dedicated host toolchain, checksum-verified build, real-container acceptance and shared contracts ([#67](https://github.com/tappe9/simple-api-benchmark/issues/67)). Outside the active official cohort; no Java result is published. |
| Versioned eight-stack cohort | Active: `eight-stack-v1`; the historical `four-stack-v1` remains frozen. [#50](https://github.com/tappe9/simple-api-benchmark/issues/50), [#57](https://github.com/tappe9/simple-api-benchmark/pull/57), [#58](https://github.com/tappe9/simple-api-benchmark/pull/58). |
| Verified eight-stack publication | Complete: the first full result was published on September 15, 2026, by [run 34925168324](https://github.com/tappe9/simple-api-benchmark/actions/runs/34925168324). See [#50](https://github.com/tappe9/simple-api-benchmark/issues/50) for measured-source, publication and Pages identities. Activation alone is not measurement evidence. |
| Result presentation | Complete: README charts, the filterable light/dark Pages dashboard and historical-run navigation. History was delivered in [#28 / #46](https://github.com/tappe9/simple-api-benchmark/pull/46); old reports retain their own cohort and provenance. |
Expand All @@ -51,9 +52,10 @@ v0.1.0 release. A next release version remains an explicit maintainer decision.

## Future candidates

Java / Spring Boot and C# / ASP.NET Core may be evaluated one at a time. These are
candidates, not commitments or implemented benchmark entries. Historical-result
navigation is already available, not a future candidate.
C# / ASP.NET Core may be evaluated as a separate implementation candidate, not a
commitment. Java / Spring Boot is already registered; its inclusion in a future
official cohort remains a separate decision that must assess the common warm-up
and measurement profile. Historical-result navigation is already available.

## Features intentionally deferred

Expand Down
6 changes: 6 additions & 0 deletions apps/java-spring-boot/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.git
.gradle
build
src/test
*.log
.env*
2 changes: 2 additions & 0 deletions apps/java-spring-boot/.gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
gradlew text eol=lf
gradlew.bat text eol=crlf
2 changes: 2 additions & 0 deletions apps/java-spring-boot/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.gradle/
build/
1 change: 1 addition & 0 deletions apps/java-spring-boot/.java-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
25.0.4.1+1
Loading
Loading