[hemju]

Closes #137.

Adds threadmill-store-oracle, an Oracle Database 19c+ JobStore held to the same AbstractJobStoreContractTest as the in-memory, PostgreSQL, and Redis stores, plus Spring Boot auto-configuration, soak/simulation backends, docs, and a 21c XE CI job.

What's in it

Commit	Scope
feat(store-oracle)	OracleJobStore, OracleMigrationRunner, schema, contract + regression + plan + migration tests
feat(spring-boot)	ThreadmillOracleAutoConfiguration, threadmill.store.oracle.*, threadmill.store.jdbc-type, join_transaction on Oracle
docs	Module README, docs/oracle-schema.md, configuration/transactions/operations/compatibility/release checklist, AGENTS.md, CI job on 21c XE
test(soak, simulation)	soakOracle, Oracle in soakRegression, simulateOracle (in simulate)

Design (as agreed in #137, with changes found while implementing)

• Standard JDBC only; applications supply the driver (ojdbc11 is test-only, so the published artifact carries no Oracle-licensed dependency).
• Startup gate: 19c+, AL32UTF8, NLS_COMP=BINARY, and an Oracle DataSource (clear message otherwise).
• Least privilege: CREATE TABLE, CREATE TRIGGER, quota. No CREATE TYPE, no DBMS_*, no AQ. DBA-applied DDL via emitPendingSql() / schema mode validate.
• Resumable migrations: Oracle commits DDL, so progress is recorded per statement; a crashed migration resumes, tolerating "already exists" only for the one statement whose DDL may have committed. Migrators serialize on a row lock held on a second connection. Files use SQL*Plus / terminators.
• No remote wake (as agreed): polling fallback across nodes; same-JVM wakes unchanged.

Things the real database taught me (all pinned by named regression tests)

1. Partial indexes → indexed VIRTUAL columns, not expression indexes. Under the 19c optimizer feature set, expression (FBI) indexes made every per-key top-n re-sort the key's whole range; virtual columns read index order and stop after n entries. Bonus: queries name plain columns instead of repeating CASE expressions.
2. No lateral joins. Correlated top-n laterals keep a full-range sort, and a collection-driven LATERAL … FETCH FIRST returned wrong results (0 rows) on Oracle 23.26. Per-key heads are ROWNUM top-n branches (UNION ALL), keys come from a recursive MIN loose scan, admission from EXISTS range probes.
3. ROWNUM, not FETCH FIRST, on hot paths: with 19c features FETCH FIRST became a WINDOW SORT over the range.
4. Locking queries can't take a row limit (ORA-02014) — index-ordered FOR UPDATE SKIP LOCKED cursors are fetched only as far as the page; a test proves Oracle locks only fetched rows.
5. ojdbc corrupts emoji in long CLOB binds: a directly bound string > ~16K chars had a surrogate pair split at the driver's ~32 KiB chunk boundary. Values > 8,000 chars now go through a temporary Clob (freed per statement).
6. Client locale changes ORDER BY: ojdbc derives NLS_SORT from the JVM locale (German here), so linguistic page order disagreed with binary > keyset predicates and skipped rows (recurring-task scans, idle-group reclamation). Paging now orders in binary (NLSSORT … BINARY / a RAW sort key).
7. EXPLAIN PLAN lies for RAW binds (treats them as VARCHAR2 → RAWTOHEX → full scan). OracleQueryPlanTest runs the store's exact SQL with typed binds and asserts DISPLAY_CURSOR plans.

Deviations from the issue text

• The Spring selector property is threadmill.store.jdbc-type (postgres|oracle) instead of threadmill.store.type: it only chooses between the two JDBC stores; Redis/memory keep their existing switches. Selection is property → spring.datasource.url scheme → module present (PostgreSQL on a tie, unchanged behaviour), decided in conditions rather than by DatabaseMetaData (conditions run before any bean exists). A mis-selected store still fails fast at startup.
• threadmill_metadata (unused by any store) is not created on Oracle.

Verification

• Oracle store: 157 tests green on Oracle 23ai Free (arm64) — contract (110), regression, plans, migrations, unit — with OPTIMIZER_FEATURES_ENABLE='19.1.0' and a German client locale.
• Contract suite green on Oracle 21c XE (amd64 emulation); CI adds a dedicated 21c XE job.
• Spring: 117 tests green (incl. 18 new Oracle/selection tests).
• simulateOracle: 400/400 succeeded, drained, 0 verifier violations (injected failures/timeouts retried).
• soakOracle (20 s, 20 jobs/s): verdict passed. OracleSoakTest 5/5 incl. container-pause recovery (5,000 jobs ≈ 917/s on a laptop container).
• ./gradlew check green on the final head (JDK 25).

Open / for review

• Real 19c run still owed — no free 19c image exists. Run ./gradlew :threadmill-store-oracle:test -PoracleJdbcUrl=… -PoracleUser=… -PoraclePassword=… against a disposable schema on your 19c (release checklist item).
• Oracle is not in the worker-churn / nudge process simulations or the dual-backend endurance default (soakEndurance keeps postgres,redis; -Pbackends=oracle works). productionCheck now also starts Oracle containers (store tests, simulate, soakRegression).
• The soak producer's outage classifier does not yet recognise Oracle restart errors (ORA-01033/01034); tracked as a follow-up in AGENTS.md.
• Database character set must be AL32UTF8; if production uses a legacy charset, NCLOB/NVARCHAR2 support would be a follow-up.

[hemju]

AI code review of PR #138 / issue #137. I read the full PR diff, linked issue, and repository guidance, then ran the Oracle 21c query-plan suite and two targeted local integration probes. The joined-transaction partial commit and mutex ownership findings below both reproduced on Oracle 21c; I recommend fixing them before merge.

The new Oracle module keeps Spring out of the store and reuses the shared JobStore contract, which fits the project architecture. The 21c CI plan-test failure also needs a stable fix. The documented 19c support still needs the planned real-19c qualification run; 21c/23ai coverage and 19.1 optimizer mode do not exercise 19c server behavior. As a maintainability follow-up, consider splitting the 2,850-line OracleJobStore by concern (claim/maintenance, dedup/concurrency, cron/mutex) into package-private collaborators while keeping one store API and transaction boundary.

The current checks also show a memory soak-smoke failure and npm audit advisories. I did not establish that those two failures came from this PR, but the release gate remains red.

[hemju]

[AI reply] Triage of the AI review

All five inline findings were valid and are fixed in 63c074a, with regression rows in AGENTS.md (f98ae27). Each thread has a reply and is resolved.

Review-level points:

• Split OracleJobStore into collaborators — agreed in principle, not done in this PR. The class mirrors PostgresJobStore (≈2,700 lines, same section layout) on purpose, so the two can be compared method by method during review and when porting future fixes. Splitting only the Oracle store would break that correspondence. I suggest a follow-up that splits both JDBC stores along the same seams (claim, maintenance, dedup/concurrency, cron/mutex) behind one store API and transaction boundary.
• Real 19c qualification — agreed. 23ai/21c with OPTIMIZER_FEATURES_ENABLE='19.1.0' covers syntax and the optimizer, not 19c server behaviour. The run is a release-checklist item (docs/release-checklist.md) and listed as an open item in the PR description.
• Dependency security (npm audit: braces, postcss, source-map-js) — pre-existing, not from this PR. The nightly main runs on c7eae24 have failed with the same advisories since 2026-10-02. This PR does not change any npm dependency.
• SoakHarnessSmokeTest (memory backend) — not reproducible locally. The PR only changes which system properties that test clears; the memory backend touches no Oracle code. Two full local ./gradlew check runs passed, including this test. The CI report shows the run's progress lines but no verdict detail, so I'm treating it as a flake and checking the next CI run.

[hemju]

AI follow-up review of commits 63c074a and f98ae27: I read the complete follow-up diff and verified that the five findings from my earlier review are addressed. The joined Oracle write now rolls back its whole unit to a savepoint, with a regression that catches the batch failure and commits an earlier successful write. Null mutex release is rejected and covered by the shared store contract. The duration bind preserves nanoseconds, the plan probe identifies its cursor explicitly, and the poll-interval documentation matches the 500 ms default. I found no new code issue in these changes.

Local verification on JDK 25: Oracle 21c query-plan and JobStore contract suites, Spring Oracle transaction tests, and the new mutex contract test on memory/PostgreSQL/Redis all passed. The PR's Oracle 21c CI job is now green. The full JDK 25 build was still running at the time of this comment. Dependency security remains red due to npm audit advisories for braces, postcss-selector-parser, and source-map-js (the npm lockfile was not changed in this PR). Real Oracle 19c qualification remains listed in the release checklist.

[hemju]

[AI] 8-hour Oracle endurance run: passed

./gradlew :threadmill-soak:soakEndurance -Pbackends=oracle (defaults: mixed workload, 50 jobs/s, 3 nodes, node churn every 10 min) on head f98ae27, against an Oracle 23ai Free container on one laptop. A 90-second pre-flight passed first.

Measure	Result
Wall clock	480 min
Enqueued / succeeded	1,440,017 / 1,440,017 (fully drained)
Failed / retried	1 / 1 — an orphan reclaim after a node's heartbeat expired; the retry succeeded
Node churn cycles	47
Invariants (atLeastOnce, exclusivityHeld, strictInGroupOrder, noLockLeaks, retryBudgetRespected)	all passed, 0 violations
End-to-end latency	p50 32 ms · p95 185 ms · p99 661 ms · max 10.2 s
Throughput	50.0 jobs/s overall, even across all 20 queues

Latency and throughput come from a laptop container, so they compare runs rather than measure production capacity. This run does not replace the real Oracle 19c test run, which is still open.