Framework Benchmarks is an extensible, cross-language project for comparing application frameworks under realistic resource constraints. It measures readiness, memory consumption, throughput, and latency with equivalent applications, shared workloads, and documented environments. Correctness checks establish comparable behavior before measurement.
The accepted v1 comparison is the project's first benchmark. Avaje Jex + CoilBase, Quarkus REST + CoilBase, and Spring Boot + Spring JDBC implement the same Market Replay Portfolio application and share a PostgreSQL schema, replay fixture, correctness contract, and measurement protocol.
Market Replay Portfolio simulates an investment portfolio service with portfolio reads and updates, valuations, and periodic market price ingestion. These operations exercise HTTP, database access, transactions, and background processing.
The runs used a Linux host (AMD Ryzen 7 4700U) running a QEMU/KVM Ubuntu VM
with 1 vCPU, 512 MiB RAM, and no swap. The application and PostgreSQL shared
the VM. Both rounds used OpenJDK 27 (build 27+35-2325) and PostgreSQL
18.7 (Ubuntu package 18.7-1.pgdg24.04+2). Sustained load was offered at 40 RPS.
| Target | HTTP framework | Persistence | Connection pool | PostgreSQL JDBC | Artifact manifests |
|---|---|---|---|---|---|
| Jex + CoilBase | Avaje Jex 3.6, Avaje HTTP 3.12.1 | CoilBase Core and CoilBase Jex adapter (both 0.1.0-preview5) | HikariCP 7.0.2 | 42.7.8 | R1, R2 |
| Quarkus + CoilBase | Quarkus REST 3.40.1 | CoilBase Core 0.1.0-preview5 | Agroal 3.2.1 | 42.7.13 | R1, R2 |
| Spring Boot + JDBC | Spring Boot 4.1.1, Spring MVC 7.0.9, Tomcat 11.0.24 | Spring JDBC 7.0.9 | HikariCP 7.0.2 | 42.7.13 | R1, R2 |
Spring Boot + Spring JDBC is the reference implementation. This benchmark
compares lightweight, SQL-first applications with equivalent functionality.
Quarkus REST handles HTTP, Quarkus supplies JDBC connectivity, and Agroal
manages the connection pool. The Quarkus target uses only coilbase-core
from CoilBase, with Jdbi for SQL access and explicit transaction management.
Neither coilbase-jex nor Avaje Jex is part of this target. We expect
CoilBase itself to add negligible overhead, but have not measured it separately.
In the earlier Hibernate isolation experiment, the same demo app used either Jdbi or direct Hibernate without Spring Data JPA. Under the mixed workload, the Jdbi implementation used about 55–58 MiB less process RSS (95.2–96.4 MiB versus 151.8–153.6 MiB). That is an application-level difference, not an isolated measurement of Hibernate's library overhead. It motivates keeping Hibernate ORM outside the current SQL-first comparison.
Readers interested in Hibernate or another persistence approach are welcome to reproduce the benchmark and submit a proposal or PR with supporting measurements. Additions should answer a specific comparison question.
Readiness is time to the first successful database-backed readiness check. RSS is the sampled peak for the application during sustained measurement. Sustained p99 includes client scheduling and SSH transport.
See the full HTML report, recorded evidence, and frozen v1 protocol.
The maintainer manually selects the featured benchmark from local or CI runs. CI provides independent reproduction and verification. Its results do not automatically replace the featured measurements.
Each target is one evolving implementation. An improvement such as Spring Boot AOT can replace its current configuration within the same target. Historical results stay tied to the exact component versions and configuration used.
The accepted v1 runs used historical market prices. The public checkout uses synthetic prices with the same symbols, sample count, and replay cadence. The public evidence omits historical prices and derived valuations from the post-load portfolio snapshots. The accepted performance measurements are unchanged. Runs with the public fixture produce separate measurements.
The archived evidence retains slot observations, memory samples, pressure counters, and artifact hashes. Host identifiers and local paths were redacted, and infrastructure captures unrelated to report regeneration were omitted.
Run commands from the checkout root. Build with JDK 25 and Maven 3.9:
mvn -B -f market-replay-portfolio/app/pom.xml package
mvn -B -f framework-benchmarks/quarkus-coilbase/pom.xml package
mvn -B -f framework-benchmarks/jex-coilbase/pom.xml package
python3 -m unittest discover -s framework-benchmarks/testsGitHub Actions runs on pushes and pull requests. It builds all three applications on Ubuntu with Temurin 25.0.4.1 and Maven 3.9.9, runs the available Java tests, checks the Linux distributions, and runs the fixture, workload and harness tests. It also compares a freshly generated offline report byte for byte with the accepted HTML.
Use an x86-64 Linux host with working /dev/kvm, QEMU, Lima 2.2+, OpenSSH,
Python 3, Bash and standard archive and checksum tools. The pinned Ubuntu image
allocates one vCPU and 512 MiB. Setup disables swap and update timers, installs
PostgreSQL 18, and verifies the pinned OpenJDK archive before extraction.
tools/vm.sh create fb-512 framework-benchmarks/vm-linux.yaml
tools/vm.sh start fb-512
curl -fL -o fb-jdk.tar.gz https://download.java.net/java/GA/jdk27/55ce5470a6294008af0057ff4626d0e5/35/GPL/openjdk-27_linux-x64_bin.tar.gz
printf '%s %s\n' 95fc37eb3a18a27a26d5904c2d89d52bace8dafa9a078ca27f4747fbc4bf070b fb-jdk.tar.gz | sha256sum -c -
tools/vm.sh put fb-512 fb-jdk.tar.gz /tmp/fb-jdk.tar.gz
tools/vm.sh put fb-512 framework-benchmarks/scripts /tmp/fb-setup
tools/vm.sh exec fb-512 sudo bash /tmp/fb-setup/setup-vm.shThe v1 PostgreSQL profile adapts Theory A from the PostgreSQL 18 low-memory experiment to leave more memory for the application sharing the VM. That experiment used a read-only workload with no colocated application. Here, PostgreSQL allows 10 connections instead of five, including one reserved superuser slot. See the v1 PostgreSQL settings.
Use the dedicated disposable database created above. The checks reset its tables and temporarily change its availability. Start PostgreSQL and leave the SSH tunnel running in a separate terminal:
tools/vm.sh exec fb-512 sudo systemctl start postgresql@18-main
tools/vm.sh tunnel fb-512 15432 5432Run the checks sequentially, with a new output path each time:
python3 framework-benchmarks/scripts/verify-postgres.py --output work/spring
python3 framework-benchmarks/scripts/verify-postgres.py --quarkus --output work/quarkus
python3 framework-benchmarks/scripts/verify-postgres.py --jex --output work/jex
tools/vm.sh exec fb-512 sudo systemctl stop postgresql@18-mainRun performance reproductions manually on a dedicated Linux QEMU/KVM host with stable resources for the guest, replay server, and load driver. Use the VM and runtime settings in the frozen v1 protocol. Check the installed PostgreSQL package before reproducing the accepted environment. The setup script installs PostgreSQL 18 from PGDG, whose default package can change. The following check must pass for the accepted package version.
test "$(tools/vm.sh exec fb-512 dpkg-query -W -f='${Version}' postgresql-18)" = 18.7-1.pgdg24.04+2Build fresh packages on Linux and pass the PostgreSQL correctness checks above
before measuring. Keep the preparation and measured outputs under separate
new paths in work/. Review the new suite against the protocol's acceptance
criteria. The accepted report and measurements remain in
framework-benchmarks/evidence/v1-accepted/.
Create three bundle directories. Each includes colocated-guest.py from
framework-benchmarks/scripts/ and benchmark-preflight.py from tools/.
Copy the Jex JAR and its adjacent lib/ together. Copy the entire Quarkus
quarkus-app/ directory. For Spring, extract the Boot layout with
java -Djarmode=tools -jar market-replay-portfolio/app/target/market-replay-portfolio-0.1.0-SNAPSHOT.jar extract --destination work/spring-bundle.
Name the extracted launcher application.jar and keep its adjacent libraries.
Use tools/prepare.sh fb-512 LOCAL_BUNDLE GUEST_BUNDLE NEW_CAPTURE_DIR for
each bundle. The following command checks both rounds with short windows.
Use a fresh output directory and omit --preparation for measurement.
python3 framework-benchmarks/scripts/colocated-suite.py run --vm fb-512 \
--protocol v1 --preparation --output work/preparation \
--bundle jex-coilbase=/tmp/fb-jex --bundle quarkus-coilbase=/tmp/fb-quarkus \
--bundle spring-boot=/tmp/fb-springpython3 framework-benchmarks/scripts/colocated-suite.py report framework-benchmarks/evidence/v1-accepted/suite.jsonThe report uses Python's standard library and writes HTML beside the manifest.
The project will support additional framework implementations, runtimes, languages, and benchmark configurations (flavors). Possible comparisons include Helidon, additional Quarkus configurations, OpenJ9, and GraalVM Native Image. How flavors will be organized and selected is still open.