Skip to content
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- CI correctness smoke coverage for the asyncpg prototype and rollback semantics
- Architecture decision retaining synchronous SQLAlchemy for v1.3.0 until
representative measurements justify a separately scoped migration
- Guarded k6 load-testing profiles for health baselines, authenticated reads,
and isolated per-virtual-user refresh-token rotation
- Explicit workload error-rate and p95/p99 starting thresholds plus a staged,
repeatable measurement guide for controlled environments

## [1.2.0] - 2026-08-09

Expand Down
88 changes: 88 additions & 0 deletions LOAD_TESTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Load testing

The repository includes a bounded [Grafana k6](https://grafana.com/docs/k6/latest/)
example for end-to-end HTTP behavior. It complements the database-only harness
in [DATABASE_BENCHMARKS.md](DATABASE_BENCHMARKS.md): k6 measures routing, JWT,
password hashing, database access, middleware, and response serialization as one
system.

## Safety and scope

Run the example only against an isolated local or explicitly authorized staging
environment. It creates disposable users and refresh-token sessions and does not
delete them. Use a disposable database or restore a snapshot after the run.

Remote targets are rejected unless `ALLOW_REMOTE_TARGET=true`. That override is
an acknowledgement, not a safety guarantee. Never point this script at
production, and confirm the expected traffic with the environment owner before
raising concurrency.

The default thresholds are reviewable starting points, not universal SLOs:

- fewer than 1% failed workload requests;
- more than 99% successful checks;
- workload p95 below 500 ms and p99 below 1,000 ms.

Change thresholds only from measured service objectives. Retain the k6 summary,
application metrics, PostgreSQL/Redis metrics, CPU, memory, commit SHA, and
environment configuration with every result.

## Profiles

| Profile | Behavior | Intended use |
| --- | --- | --- |
| `health` | Repeated `GET /health/live` after a readiness probe | Network and middleware baseline |
| `authenticated` | One user/session per VU, repeated `GET /auth/me`, periodic refresh rotation | Representative authentication lifecycle |

Registration and login happen outside the tagged workload latency thresholds.
They are still checked and visible in the k6 result. Each virtual user owns its
refresh token so rotations do not create artificial replay failures.

## Local smoke run

Start PostgreSQL and Redis, apply migrations, and start the API as described in
[DEVELOPMENT.md](DEVELOPMENT.md). Install k6 1.x, then run:

```bash
k6 run -e PROFILE=health -e VUS=2 -e DURATION=10s load_tests/api.js
k6 run -e PROFILE=authenticated -e VUS=5 -e DURATION=30s load_tests/api.js
```

PowerShell uses the same commands. The defaults target
`http://127.0.0.1:8000`, use five virtual users for 30 seconds, and rotate each
virtual user's refresh token every 20 iterations.

Useful environment variables:

| Variable | Default | Meaning |
| --- | --- | --- |
| `BASE_URL` | `http://127.0.0.1:8000` | API origin without a trailing slash |
| `PROFILE` | `authenticated` | `health` or `authenticated` |
| `VUS` | `5` | Concurrent virtual users |
| `DURATION` | `30s` | k6 duration expression |
| `REFRESH_EVERY` | `20` | Authenticated iterations between rotations |
| `RUN_ID` | current timestamp | Suffix making generated usernames unique |
| `LOAD_TEST_PASSWORD` | disposable local value | Password for generated users |
| `ALLOW_REMOTE_TARGET` | unset | Must be exactly `true` for a remote host |

Validate the script without sending traffic:

```bash
k6 inspect load_tests/api.js
```

## Staged workload

Establish a baseline before looking for capacity limits:

1. Run `health` with one VU to record network/middleware latency.
2. Run `authenticated` at 1, 5, 10, and 25 VUs for at least five minutes each.
3. Repeat every point three times on the same commit and unchanged environment.
4. Stop when error rate, pool wait, CPU saturation, or tail latency violates the
service objective; do not keep increasing traffic through an unhealthy API.
5. Diagnose bcrypt cost, database pool pressure, Redis latency, worker count,
and tracing export separately before changing the application architecture.

CI shared runners should only validate correctness and script structure. Do not
use their timings as a release performance gate. A release gate belongs in a
controlled staging environment with stable infrastructure and retained results.
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ contributor workflows, and local troubleshooting.
| [API examples](API_EXAMPLES.md) | Register, authenticate, rotate tokens, call admin routes, and inspect operations endpoints |
| [Architecture](ARCHITECTURE.md) | Understand module boundaries, request flow, authentication, transactions, and extension points |
| [Database benchmarks](DATABASE_BENCHMARKS.md) | Reproduce sync/async PostgreSQL comparisons and understand the v1.3 architecture decision |
| [Load testing](LOAD_TESTING.md) | Exercise health and authenticated HTTP lifecycles with bounded k6 workloads and explicit thresholds |
| [Local development](DEVELOPMENT.md) | Set up a checkout, run common commands, contribute, or troubleshoot locally |
| [Deployment](DEPLOYMENT.md) | Configure a production host, release safely, terminate TLS, and operate the service |
| [Monitoring](MONITORING.md) | Configure probes, Prometheus, multi-worker metrics, alerts, logs, and incident diagnosis |
Expand Down Expand Up @@ -250,6 +251,10 @@ and async database benchmark and uploads its machine-readable JSON artifact.
See [DATABASE_BENCHMARKS.md](DATABASE_BENCHMARKS.md); shared-runner timings are
not used as performance gates.

End-to-end k6 examples cover a health baseline and an authenticated lifecycle
with isolated per-VU refresh rotation. See [LOAD_TESTING.md](LOAD_TESTING.md)
for safety guards, starting thresholds, and a staged workload methodology.

CI runs against PostgreSQL 17 rather than silently substituting SQLite. It also
verifies that the built wheel contains and can import the application.

Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Status: in progress
- Transactional outbox and background worker processing (completed)
- Safe OIDC discovery/JWKS caching and invalidation guidance (completed)
- Async database evaluation and performance benchmarks (completed; sync retained)
- Load-testing examples
- Load-testing examples (completed)
- OpenTelemetry tracing example (completed)

## v2.0 — Deployment patterns at scale
Expand Down
136 changes: 136 additions & 0 deletions load_tests/api.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
import http from "k6/http";
import { check, fail, sleep } from "k6";

const BASE_URL = (__ENV.BASE_URL || "http://127.0.0.1:8000").replace(/\/$/, "");
const PROFILE = __ENV.PROFILE || "authenticated";
const VUS = positiveInteger("VUS", 5);
const DURATION = __ENV.DURATION || "30s";
const REFRESH_EVERY = positiveInteger("REFRESH_EVERY", 20);
const PASSWORD = __ENV.LOAD_TEST_PASSWORD || "local-load-test-password";
const RUN_ID = (__ENV.RUN_ID || `${Date.now()}`)
.replace(/[^a-zA-Z0-9_-]/g, "")
.slice(0, 40);

if (!["health", "authenticated"].includes(PROFILE)) {
throw new Error("PROFILE must be health or authenticated");
}

assertSafeTarget(BASE_URL);

export const options = {
vus: VUS,
duration: DURATION,
discardResponseBodies: true,
thresholds: {
checks: ["rate>0.99"],
"http_req_failed{phase:workload}": ["rate<0.01"],
"http_req_duration{phase:workload}": ["p(95)<500", "p(99)<1000"],
},
};

function positiveInteger(name, fallback) {
const value = Number.parseInt(__ENV[name] || `${fallback}`, 10);
if (!Number.isInteger(value) || value < 1) {
throw new Error(`${name} must be a positive integer`);
}
return value;
}

function assertSafeTarget(value) {
const parsed = value.match(/^https?:\/\/(\[[^\]]+\]|[^/:]+)(?::\d+)?(?:\/.*)?$/i);
if (!parsed) {
throw new Error("BASE_URL must be an absolute HTTP(S) URL");
}
const hostname = parsed[1].toLowerCase();
const localHosts = new Set(["localhost", "127.0.0.1", "[::1]", "host.docker.internal"]);
if (!localHosts.has(hostname) && __ENV.ALLOW_REMOTE_TARGET !== "true") {
throw new Error(
"Refusing a remote target. Set ALLOW_REMOTE_TARGET=true only for an authorized non-production environment.",
);
}
}

function workloadTags(endpoint) {
return { tags: { phase: "workload", endpoint } };
}

export function setup() {
const ready = http.get(`${BASE_URL}/health/ready`, workloadTags("readiness"));
if (!check(ready, { "target is ready": (response) => response.status === 200 })) {
fail(`Target is not ready: HTTP ${ready.status}`);
}

if (PROFILE === "health") {
return { users: [] };
}

const users = [];
for (let index = 1; index <= VUS; index += 1) {
const username = `load_${RUN_ID}_${index}`;
const registration = http.post(
`${BASE_URL}/register/`,
JSON.stringify({ username, password: PASSWORD }),
{ headers: { "Content-Type": "application/json" }, tags: { phase: "setup", endpoint: "register" } },
);
if (!check(registration, { "load-test user registered": (response) => response.status === 200 })) {
fail(`Could not register ${username}: HTTP ${registration.status}`);
}
users.push(username);
}
return { users };
}

let tokens;

function login(username) {
const response = http.post(
`${BASE_URL}/login/`,
{ username, password: PASSWORD },
{
headers: { "X-Device-Name": `k6-vu-${__VU}` },
tags: { phase: "setup", endpoint: "login" },
responseType: "text",
},
);
if (!check(response, { "load-test user logged in": (result) => result.status === 200 })) {
fail(`Could not log in ${username}: HTTP ${response.status}`);
}
return response.json();
}

function authenticatedIteration(data) {
if (!tokens) {
tokens = login(data.users[__VU - 1]);
}

const me = http.get(`${BASE_URL}/auth/me`, {
headers: { Authorization: `Bearer ${tokens.access_token}` },
...workloadTags("auth_me"),
});
check(me, { "authenticated read succeeds": (response) => response.status === 200 });

if (__ITER > 0 && __ITER % REFRESH_EVERY === 0) {
const rotated = http.post(
`${BASE_URL}/auth/refresh`,
JSON.stringify({ refresh_token: tokens.refresh_token }),
{
headers: { "Content-Type": "application/json" },
...workloadTags("refresh"),
responseType: "text",
},
);
if (check(rotated, { "refresh rotation succeeds": (response) => response.status === 200 })) {
tokens = rotated.json();
}
}
}

export default function (data) {
if (PROFILE === "health") {
const response = http.get(`${BASE_URL}/health/live`, workloadTags("liveness"));
check(response, { "liveness succeeds": (result) => result.status === 200 });
} else {
authenticatedIteration(data);
}
sleep(0.1);
}
2 changes: 2 additions & 0 deletions tests/test_documentation.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
"CODE_OF_CONDUCT.md",
"CONTRIBUTING.md",
"DATABASE_BENCHMARKS.md",
"LOAD_TESTING.md",
"DEPLOYMENT.md",
"DEVELOPMENT.md",
"MONITORING.md",
Expand Down Expand Up @@ -89,6 +90,7 @@ def test_readme_indexes_the_task_guides():
"API_EXAMPLES.md",
"ARCHITECTURE.md",
"DATABASE_BENCHMARKS.md",
"LOAD_TESTING.md",
"DEPLOYMENT.md",
"DEVELOPMENT.md",
"MONITORING.md",
Expand Down