Replace inline flush with a durable worker-owned job queue - #86
Merged
Conversation
Enqueue-and-return flush_table, session table locks with attempt fencing, and one-shot executors keep slot locks off the upload path. Add typed failpoints, fault/model stores, and catalog integrity checks while tightening SeqId/TableOid usage and pass-scoped memory. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
E2E (PG 16)182 tests +15 182 ✅ +15 11m 33s ⏱️ + 7m 28s Results for commit 5307013. ± Comparison against base commit 5a56e16. This pull request removes 19 and adds 34 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
E2E (PG 18)179 tests +12 179 ✅ +12 12m 3s ⏱️ + 8m 6s Results for commit 75f87cb. ± Comparison against base commit 5a56e16. This pull request removes 19 and adds 31 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
E2E (PG 17)180 tests +13 180 ✅ +13 11m 49s ⏱️ + 7m 55s Results for commit 1ad22d8. ± Comparison against base commit 5a56e16. This pull request removes 19 and adds 32 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
E2E (PG 15)182 tests +15 182 ✅ +15 9m 23s ⏱️ + 5m 13s Results for commit 5307013. ± Comparison against base commit 5a56e16. This pull request removes 19 and adds 34 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Keep apply live during Parquet upload, document the short exclusive finalize window, and add e2e coverage for ≤1s mirror/changes_since visibility with auto_flush off plus manual flush probes. Co-authored-by: Cursor <cursoragent@cursor.com>
Fix Short-txn SPI and advisory unlock crashes, invalidate cold cache before hot prune, and expand default-loop tests for queue flush, peak RSS, startup reclaim, and JSON EXPLAIN tracing. Co-authored-by: Cursor <cursoragent@cursor.com>
pg_test (PG 15)89 tests +7 89 ✅ +7 2m 42s ⏱️ -27s Results for commit 155a929. ± Comparison against base commit 5a56e16. This pull request removes 1 and adds 8 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
pg_test (PG 18)89 tests +7 89 ✅ +7 3m 6s ⏱️ +15s Results for commit 155a929. ± Comparison against base commit 5a56e16. This pull request removes 1 and adds 8 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
pg_test (PG 16)89 tests +7 89 ✅ +7 2m 46s ⏱️ -9s Results for commit 155a929. ± Comparison against base commit 5a56e16. This pull request removes 1 and adds 8 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
pg_test (PG 17)89 tests +7 89 ✅ +7 2m 47s ⏱️ -33s Results for commit 155a929. ± Comparison against base commit 5a56e16. This pull request removes 1 and adds 8 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Emit Internal pipeline nodes under KoldStore Pipeline so they do not collide with PostgreSQL's Plans key after JSON parse. Co-authored-by: Cursor <cursoragent@cursor.com>
Add extension lifecycle and mid-read cancel e2e (plus Toxiproxy cold-read), join JSON EXPLAIN tracing, and peak-memory runner wiring; drop tautological matrix stubs and document why shell-tests stay a sibling crate. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Require selected excess to meet max_rows_per_file before enqueueing, set progress_total from the policy selection instead of the full mirror count, drop progress_unit, and deepen live/job failure coverage. Co-authored-by: Cursor <cursoragent@cursor.com>
Wire Parquet sorting metadata from the flush encode path and format the files that were failing cargo fmt on CI. Co-authored-by: Cursor <cursoragent@cursor.com>
Run pg/async sides on separate runners, compare against the prior successful CI baseline, and post one updated markdown table comment per PR. Co-authored-by: Cursor <cursoragent@cursor.com>
When half-chunk rounding drops selected rows below max_rows_per_file but excess still fills a file, flush one full file. Retry failpoint recovery with force so late cleanup failpoints do not treat a no-op policy flush as failure. Co-authored-by: Cursor <cursoragent@cursor.com>
Fence the async mirror before policy flush helpers so concurrent writers are visible to the due check, size the orphan-reclaim pg_test so excess meets max_rows_per_file, and download storage-bench artifacts by explicit name. Co-authored-by: Cursor <cursoragent@cursor.com>
Treat policy flush NULL as success, size scheduler fixtures under the file floor, seed three_state before manage, and write bench JSON outside Cargo target/ so upload-artifact can find it. Co-authored-by: Cursor <cursoragent@cursor.com>
…h default. Resolve storage-bench JSON to an absolute path for nextest cwd, force pending flush enqueue/drop cleanup, and default sqlreg.flush_table to force so policy no-ops keep stable UUID expected output. Co-authored-by: Cursor <cursoragent@cursor.com>
Storage bench (PG 16, 10k rows)No prior successful CI storage-bench artifacts found; showing current run only (deltas marked n/a). Results for commit
♻️ This comment has been updated with latest results. |
…alamdb/koldstore into feature/worker-flush-job-queue
sqlreg.flush_table now defaults to force; demigrate needs a non-force flush to emit one policy-sized segment before unmanage. Co-authored-by: Cursor <cursoragent@cursor.com>
…h runs. manage_with_hot_limit used max_rows_per_file=5000, so small excess never reached failpoint waits after the undersized-segment gate. Co-authored-by: Cursor <cursoragent@cursor.com>
Adds a new weekly-long-tests GitHub workflow for slow/gated suites (full crash matrix, postmaster restart, toxiproxy, soak, long SQLsmith/differential runs) and updates nightly crash readiness to always cover executor SIGKILL paths. Flush execution now has a configurable wall-clock budget via `koldstore.flush_job_max_runtime_seconds` (default 1800s), enforced during passes and streamed chunks to prevent runaway jobs. Queue job handling is tightened to reclaim ownerless running jobs and upgrade pending jobs when force flush is requested. Failpoint handling is also safer and cheaper: production builds reject wait/panic/sleep actions, wait barriers are interruptible, and hot-path failpoint reads avoid extra string allocations. E2E crash/flush tests were expanded and refactored with shared recovery invariants, including new long-flush SIGKILL data-plane coverage and harness improvements.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+58
to
+94
| name: Resolve PostgreSQL matrix | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| pg: ${{ steps.set.outputs.pg }} | ||
| steps: | ||
| - id: set | ||
| name: Build PG major JSON array | ||
| run: | | ||
| set -euo pipefail | ||
| if [[ "${{ github.event_name }}" == "schedule" ]]; then | ||
| echo 'pg=[16]' >> "$GITHUB_OUTPUT" | ||
| echo "schedule → PostgreSQL [16]" | ||
| exit 0 | ||
| fi | ||
|
|
||
| versions=() | ||
| [[ "${{ inputs.pg15 }}" == "true" ]] && versions+=(15) | ||
| [[ "${{ inputs.pg16 }}" == "true" ]] && versions+=(16) | ||
| [[ "${{ inputs.pg17 }}" == "true" ]] && versions+=(17) | ||
| [[ "${{ inputs.pg18 }}" == "true" ]] && versions+=(18) | ||
|
|
||
| if [[ ${#versions[@]} -eq 0 ]]; then | ||
| echo "error: select at least one PostgreSQL version (15–18)" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Compact JSON array without requiring jq. | ||
| json="[" | ||
| for i in "${!versions[@]}"; do | ||
| [[ "$i" -gt 0 ]] && json+="," | ||
| json+="${versions[$i]}" | ||
| done | ||
| json+="]" | ||
| echo "pg=${json}" >> "$GITHUB_OUTPUT" | ||
| echo "workflow_dispatch → PostgreSQL ${json}" | ||
|
|
||
| long-tests: |
Comment on lines
+95
to
+233
| name: Long tests (PG ${{ matrix.pg }}) | ||
| needs: resolve-matrix | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 240 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| pg: ${{ fromJSON(needs.resolve-matrix.outputs.pg) }} | ||
| env: | ||
| PG_VER: ${{ matrix.pg }} | ||
| KOLDSTORE_E2E_PGVERSION: ${{ matrix.pg }} | ||
| KOLDSTORE_E2E_PGPORT: ${{ format('288{0}', matrix.pg) }} | ||
| KOLDSTORE_E2E_PGHOST: 127.0.0.1 | ||
| KOLDSTORE_E2E_PGDATABASE: koldstore_pgrx_e2e | ||
| KOLDSTORE_E2E_WAIT_FOR_STARTUP: "1" | ||
| KOLDSTORE_PGRX_INSTALL_SUDO: "1" | ||
| PGRX_PG_CONFIG: /usr/lib/postgresql/${{ matrix.pg }}/bin/pg_config | ||
| KOLDSTORE_CRASH_FLUSH_EXECUTOR: "1" | ||
| KOLDSTORE_SQLSMITH_SECONDS: ${{ inputs.sqlsmith_seconds || '300' }} | ||
| KOLDSTORE_DIFF_SQLSMITH_SECONDS: ${{ inputs.sqlsmith_seconds || '300' }} | ||
| KOLDSTORE_MINIO: "1" | ||
| KOLDSTORE_MINIO_ENDPOINT: http://127.0.0.1:9000 | ||
| KOLDSTORE_MINIO_ACCESS_KEY: minioadmin | ||
| KOLDSTORE_MINIO_SECRET_KEY: minioadmin | ||
| KOLDSTORE_MINIO_BUCKET: koldstore-test | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - uses: dtolnay/rust-toolchain@1.96.0 | ||
| - uses: Swatinem/rust-cache@v2 | ||
| with: | ||
| # Separate caches per PG major (different feature builds). | ||
| key: long-tests-pg${{ matrix.pg }} | ||
| - uses: taiki-e/install-action@v2 | ||
| with: | ||
| tool: nextest | ||
|
|
||
| - name: Install PostgreSQL ${{ matrix.pg }} development packages | ||
| run: bash scripts/ci/install-pgdg-dev.sh "${PG_VER}" | ||
|
|
||
| - name: Install cargo-pgrx | ||
| run: cargo install cargo-pgrx --version ${PGRX_VERSION} --locked | ||
|
|
||
| - name: Initialize pgrx for PostgreSQL ${{ matrix.pg }} | ||
| run: cargo pgrx init --pg${PG_VER} "${PGRX_PG_CONFIG}" | ||
|
|
||
| - name: Start MinIO | ||
| run: bash scripts/ci/start-minio.sh | ||
|
|
||
| - name: Start Toxiproxy | ||
| run: bash scripts/ci/start-toxiproxy.sh | ||
|
|
||
| - name: Prepare pgrx cluster + install koldstore | ||
| run: bash scripts/run-pg-e2e.sh "${PG_VER}" | ||
| env: | ||
| KOLDSTORE_E2E_PREPARE_ONLY: "1" | ||
|
|
||
| - name: Crash recovery (executor SIGKILL + failpoints) | ||
| run: | | ||
| set -euo pipefail | ||
| # shellcheck disable=SC1091 | ||
| source "${KOLDSTORE_E2E_ENV_FILE:-.e2e-env}" | ||
| export KOLDSTORE_CRASH_FLUSH_EXECUTOR=1 | ||
| # Schedule always runs the full matrix; dispatch honors the input. | ||
| if [[ "${{ github.event_name }}" == "schedule" || "${{ inputs.full_crash_matrix }}" == "true" ]]; then | ||
| export KOLDSTORE_CRASH_FULL_MATRIX=1 | ||
| fi | ||
| cargo nextest run -p e2e \ | ||
| -E 'test(crash::) & not test(postmaster_restart::)' \ | ||
| --test-threads "${KOLDSTORE_E2E_THREADS:-1}" | ||
|
|
||
| - name: Postmaster immediate restart mid-flush | ||
| run: | | ||
| set -euo pipefail | ||
| # shellcheck disable=SC1091 | ||
| source "${KOLDSTORE_E2E_ENV_FILE:-.e2e-env}" | ||
| export KOLDSTORE_CRASH_POSTMASTER_RESTART=1 | ||
| cargo nextest run -p e2e \ | ||
| -E 'test(crash::postmaster_restart::)' \ | ||
| --test-threads 1 | ||
|
|
||
| - name: Toxiproxy network-fault + cold-read cancel | ||
| run: | | ||
| set -euo pipefail | ||
| # shellcheck disable=SC1091 | ||
| source "${KOLDSTORE_E2E_ENV_FILE:-.e2e-env}" | ||
| export KOLDSTORE_TOXIPROXY=1 | ||
| export KOLDSTORE_MINIO=1 | ||
| cargo nextest run -p e2e \ | ||
| -E 'test(failure_injection::) + test(query_cancel::)' \ | ||
| --test-threads 1 | ||
|
|
||
| - name: Optional soak | ||
| if: ${{ github.event_name == 'schedule' || inputs.soak }} | ||
| run: | | ||
| set -euo pipefail | ||
| # shellcheck disable=SC1091 | ||
| source "${KOLDSTORE_E2E_ENV_FILE:-.e2e-env}" | ||
| export KOLDSTORE_E2E_SOAK=1 | ||
| export KOLDSTORE_E2E_SOAK_SECONDS="${KOLDSTORE_E2E_SOAK_SECONDS:-45}" | ||
| cargo nextest run -p e2e -E 'test(endurance::)' --test-threads 1 | ||
|
|
||
| - name: Install SQLsmith | ||
| run: bash scripts/ci/install-sqlsmith.sh | ||
|
|
||
| - name: SQLsmith (long fuzz) | ||
| run: bash scripts/readiness/run-sqlsmith.sh "${PG_VER}" | ||
| env: | ||
| SQLSMITH_BIN: ${{ github.workspace }}/target/tools/sqlsmith/bin/sqlsmith | ||
|
|
||
| - name: Differential SQLsmith compare (long) | ||
| run: bash scripts/readiness/run-differential-sqlsmith.sh "${PG_VER}" | ||
| env: | ||
| KOLDSTORE_DIFF_STATE: mixed | ||
| SQLSMITH_BIN: ${{ github.workspace }}/target/tools/sqlsmith/bin/sqlsmith | ||
|
|
||
| - name: Integrity checks | ||
| run: bash scripts/readiness/run-integrity-checks.sh "${PG_VER}" | ||
| env: | ||
| KOLDSTORE_INTEGRITY_PREPARE: "0" | ||
|
|
||
| - name: Emit long-tests report | ||
| if: always() | ||
| run: | | ||
| export KOLDSTORE_REPORT_CATEGORY=weekly-long-tests-pg${PG_VER} | ||
| export KOLDSTORE_REPORT_PASSED=true | ||
| export KOLDSTORE_REPORT_SQLSMITH=ran | ||
| export KOLDSTORE_REPORT_DIFFERENTIAL=ran | ||
| export KOLDSTORE_REPORT_CRASH=ran | ||
| export KOLDSTORE_REPORT_POSTMASTER_RESTART=ran | ||
| export KOLDSTORE_REPORT_INTEGRITY=ran | ||
| bash scripts/readiness/run-readiness-report.sh "${PG_VER}" | ||
|
|
||
| - name: Upload long-tests artifacts | ||
| uses: actions/upload-artifact@v7 | ||
| if: always() | ||
| with: | ||
| name: weekly-long-tests-report-pg${{ matrix.pg }} | ||
| path: target/readiness | ||
| if-no-files-found: warn |
Open
68 tasks
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
flush_tableenqueue-and-return a job UUID; coordinators spawn one-shot flush executors with session table locks and attempt fencing so slot locks stay off the object upload path.koldstore-flush, fault-injecting + in-memory model object stores inkoldstore-storage, andkoldstore.verify_table_integritycatalog checks.SeqId/TableOid/TableNameusage, drop legacy inline-job / abandon APIs, and keep pass-scoped flush memory bounded (segment refs only after catalog insert; capped fault traces and integrity samples).Test plan
cargo test -p koldstore-flush --libcargo test -p koldstore-storage --libcargo test -p koldstore-catalog --libcargo test -p koldstore-worker --libcargo check -p pg_koldstore --features "pg17,s3"pg_testpath withkoldstore.flush_execution = 'inline'Made with Cursor