diff --git a/.github/workflows/benchmarks.yml b/.github/workflows/benchmarks.yml index 29c5183..25b7fb9 100644 --- a/.github/workflows/benchmarks.yml +++ b/.github/workflows/benchmarks.yml @@ -37,7 +37,7 @@ jobs: - name: HS256 smoke regression (same gate as main CI) run: .venv/bin/python -m pytest tests/test_benchmark_jwt_libraries.py::test_benchmark_hs256_smoke_vs_competitors -q - - name: Run full comparison script + - name: Run full comparison script (default pem keys) run: | mkdir -p benchmark-results .venv/bin/python scripts/compare_jwt_libraries.py \ @@ -47,7 +47,17 @@ jobs: --warmup 50 \ --markdown benchmark-results/ci-bench.md + - name: Run asymmetric comparison (cached competitor keys) + run: | + .venv/bin/python scripts/compare_jwt_libraries.py \ + --algorithms RS256,EdDSA \ + --iterations 200 \ + --rounds 2 \ + --warmup 50 \ + --competitor-key-mode cached \ + --markdown benchmark-results/ci-bench-cached.md + - uses: actions/upload-artifact@v4 with: name: benchmark-md - path: benchmark-results/ci-bench.md + path: benchmark-results/ diff --git a/README.md b/README.md index 6428c45..a4b6624 100644 --- a/README.md +++ b/README.md @@ -129,6 +129,8 @@ python -m venv .venv The script covers HMAC, RSA, RSA-PSS, ECDSA, and EdDSA algorithms. Unsupported library/algorithm combinations are reported as `0` throughput. For a quicker smoke test, pass something like `--algorithms HS256,RS256,EdDSA --iterations 100 --rounds 1`. +**Benchmark fairness:** the default `--competitor-key-mode pem` keeps pre-parsed `EncodingKey`/`DecodingKey` for OxyJWT while competitors often receive PEM bytes (see [Benchmarks](docs-site/docs/benchmarks.md)). For asymmetric comparisons, also run with `--competitor-key-mode cached`. + Benchmark outputs are ignored by git because results depend on the machine, Python version, compiler flags, and CPU state. The default Rust crypto backend is `aws_lc_rs`, chosen for stronger performance on RSA and ECDSA in local benchmarks. You can still build with `rust_crypto` for comparison: @@ -152,20 +154,19 @@ OxyJWT implements JWT/JWS signing and verification. JWE encryption is not part o See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and pull request expectations. Report security issues privately via [SECURITY.md](SECURITY.md). -## 🚀 Performance Benchmarks +## Performance benchmarks -OxyJWT is built for absolute speed. By bypassing the Python GIL and leveraging Rust's cryptographic primitives, it completely destroys standard Python libraries in both symmetric and asymmetric cryptography. +OxyJWT is optimized for throughput on typical JWT workloads (especially HMAC). See [docs-site/docs/benchmarks.md](docs-site/docs/benchmarks.md) for smoke vs extended vs full workflows and key-preparation modes. -Below is a performance comparison measured in **Operations per second (ops/sec)** (higher is better): +The table below is a **historical snapshot** (default script settings, `pem` competitor keys). RS256 encode numbers are not comparable to `--competitor-key-mode cached`; re-run the script on your hardware before drawing conclusions. -| Algorithm | Operation | ⚡ OxyJWT | PyJWT | Authlib | python-jose | -| :--- | :--- | :--- | :--- | :--- | :--- | -| **HS256** | Encode | **620,270** | 140,670 | 99,408 | 99,507 | -| **HS256** | Decode | **361,073** | 109,272 | 94,823 | 51,838 | -| **RS256** | Encode | **1,934** | 35 | 35 | 35 | -| **RS256** | Decode | **58,752** | 27,200 | 26,085 | 23,046 | -| **EdDSA** | Encode | **69,105** | 17,518 | 15,014 | N/A | -| **EdDSA** | Decode | **31,666** | 10,741 | 10,317 | N/A | -| **ES256** | Encode | **46,559** | 19,632 | 16,199 | 19,723 | +| Algorithm | Operation | OxyJWT | PyJWT | Authlib | python-jose | +| :--- | :--- | ---: | ---: | ---: | ---: | -*Tested against standard Python ecosystem libraries. OxyJWT consistently dominates across all algorithms.* \ No newline at end of file +| **HS256** | Encode | 620,270 | 140,670 | 99,408 | 99,507 | +| **HS256** | Decode | 361,073 | 109,272 | 94,823 | 51,838 | +| **RS256** | Encode | 1,934 | 35 | 35 | 35 | +| **RS256** | Decode | 58,752 | 27,200 | 26,085 | 23,046 | +| **EdDSA** | Encode | 69,105 | 17,518 | 15,014 | N/A | +| **EdDSA** | Decode | 31,666 | 10,741 | 10,317 | N/A | +| **ES256** | Encode | 46,559 | 19,632 | 16,199 | 19,723 | \ No newline at end of file diff --git a/scripts/compare_jwt_libraries.py b/scripts/compare_jwt_libraries.py index 8207737..cac1aa9 100644 --- a/scripts/compare_jwt_libraries.py +++ b/scripts/compare_jwt_libraries.py @@ -506,7 +506,11 @@ def parse_args() -> argparse.Namespace: "--competitor-key-mode", choices=("pem", "cached"), default="pem", - help="Use PEM bytes for competitors or preloaded cryptography key objects where supported.", + help=( + "pem (default): competitors get PEM str/bytes; OxyJWT uses pre-parsed " + "EncodingKey/DecodingKey. cached: preloaded cryptography objects for " + "fairer RSA/EC/EdDSA vs PyJWT/Authlib." + ), ) parser.add_argument( "--algorithms",