SPOOL profiles the source, infers a safe target schema, builds deterministic transforms, dry-runs them, executes in a Worker, groups bad rows, recovers from checkpoints, and exposes only the agent tools that make sense right now.
Autopilot is not a script that clicks the old UI. It is a kernel-level orchestration path that plans, dry-runs and starts the migration through the same deterministic commands used everywhere else.
Run Autopilot ${arrowIcon()}Add source↓Choose Database-ready↓Run Autopilot↓No action required${copy}
SPOOL only advances when the target and transform plan satisfy deterministic safety rules. Strong parseability evidence can promote a dirty string field to a type while rows that fail that contract are rejected into quality groups.
Whitespace trim, normalized identifiers, obvious booleans, strongly parseable numbers and ISO dates.
Name collisions, multiple equally plausible interpretations, or a plan that cannot satisfy the typed target contract.
target field: monthly_fee\ninferred type: number\nconfidence: 0.9900\nevidence: 99 / 100 sampled values parse\ndecision: automatic
The application stopped before entering the migration workflow instead of leaving you on a permanent loading screen.
${esc(message)}The application stopped before entering the migration workflow instead of leaving you on a permanent loading screen.
${esc(message)}The Browser Studio prepares and validates data in-browser. The Local Runner is the production mutation path: bounded-memory streaming of approved UTF-8 filesystem CSV migrations into an existing ordinary SQLite table on the customer-controlled machine, with a 256 MiB default source ceiling.
+The Browser Studio prepares and validates data in-browser. The released Local Runner is the customer-local mutation path: bounded-memory streaming of approved UTF-8 filesystem CSV migrations into an existing ordinary SQLite table on the customer-controlled machine, with a 256 MiB default source ceiling.
Node.js 22+ is required. Browser Studio stays capped at 50 MiB; for production SQLite mutation or larger CSV files use this customer-local runner. The Local Runner uses bounded-memory streaming with a 256 MiB default source ceiling. It keeps a durable content-bound snapshot on local disk for approval/recovery, re-scans that snapshot on restart, and cleans it after VERIFIED completion. Budget disk space for the snapshot plus SQLite/WAL growth. The release remains private on the npm registry; install directly from the versioned GitHub source or release artifact.
# Preferred v1.1.1 install — no npm-registry publication required. ++ BEFORE INSTALLINGFor a SQLite database operator
First prepare the CSV and an existing compatible SQLite table. Then install SPOOL locally, create the documented migration request, inspect and dry-run it, authorize the exact plan, execute, verify and save the receipt. The published capability matrix classifies SQLite target support as experimental because sustained-load qualification is still missing; do not interpret a release download as proof of unrestricted enterprise readiness.
Return to the guided setup →+cd spool && npm ci && npm run check && npm run pack:verify ADVANCED · INSTALL + RUN ON YOUR DEVICEInstall the versioned CLI, then inspect, approve, run and verify.
Node.js 22+ is required. Browser Studio stays capped at 50 MiB; for production SQLite mutation or larger CSV files use this customer-local runner. The Local Runner uses bounded-memory streaming with a 256 MiB default source ceiling. It keeps a durable content-bound snapshot on local disk for approval/recovery, re-scans that snapshot on restart, and cleans it after VERIFIED completion. Budget disk space for the snapshot plus SQLite/WAL growth. The release remains private on the npm registry; install directly from the versioned GitHub source or release artifact.
Show command-line installation and execution steps
# Preferred v1.1.1 install — no npm-registry publication required. npm install -g github:dharan1007/spool#v1.1.1 spool --help @@ -102,7 +101,7 @@ spool receipt --migration-id mig_customers_001 --source-root ./data --target-roo # Source-verification fallback: git clone --branch v1.1.1 https://github.com/dharan1007/spool.git -cd spool && npm ci && npm run check && npm run pack:verify
Choose the outcome first. SPOOL will show the correct interface, what runs on your device, and the evidence you will receive. You do not need to understand API responses to choose a path.
You have a CSV, and want a validated CSV or JSON output. Add the file in your browser, choose a cleanup outcome, run Autopilot, review rejected rows and export. Nothing is sent to a SPOOL data server.
Input: CSV up to 50 MiB · Result: browser-local output, not a live database write.
Open the CSV Studio →You have a CSV and an existing SQLite database. Install the released local runner on the computer holding the data. An operator reviews source and destination, runs a dry-run, approves the exact operation, executes it and examines the receipt.
Default CSV ceiling: 256 MiB · Insert-only, ordinary SQLite targets · Command line and setup required.
See the local setup →Your application can invoke SPOOL through its local CLI or an authenticated loopback bridge. A developer installs and runs the command service on the customer-controlled machine. This website does not host a remote migration API.
Execution stays local · Approvals are not bypassed · Requires development work.
Explore integration paths →Those are not part of the currently published v1.1.1 PostgreSQL production scope. Do not put production credentials into Browser Studio or assume an experimental connector is qualified.
Read capability and limit explanations →An integration is software running on your device, not a button that uploads your database to this site. Pick the supported execution surface before writing any code.
Open Studio, choose your CSV and an outcome, inspect validation, and download the result. No API key, server or integration snippet is necessary.
Open Studio →Install v1.1.1 on the machine holding your existing SQLite table and CSV. A developer can invoke the CLI within local automation, preserving the inspect → plan → dry-run → approve → run → verify → receipt sequence.
Local runner instructions →The source includes an authenticated loopback command service. It is not a hosted API and is not started merely by opening this website. Configure it with a local command service, a strong bearer token and an approved Origin policy.
Bridge contract and setup →Machine-readable command responses belong in developer tools; an HTTP endpoint does not provide a human UI. For humans, use Studio or these guides. For applications, use the local CLI or the loopback bridge after provisioning the customer-controlled runtime. This hosted website has no public database-write API, no cloud database connection form and no universal integration token.
Start with the released CLI for production-scoped filesystem CSV to SQLite operations. The loopback HTTP bridge is an integration component, not a remotely hosted SPOOL API. Code examples and machine formats are deliberately kept in the technical section below.
Install the exact released version on the same customer-controlled system as the source and target. Use a separately configured local bridge only when a client requires HTTP.
Inspect source and live target, create a plan, then run a dry-run. Stop on source or contract drift.
Issue the supported bound approval for the exact operation. An API invocation is not itself permission to mutate.
Run only the approved operation, reconcile ambiguous results, verify the supported evidence and retrieve the receipt.
The current source exports a loopback-only daemon factory. A developer must initialize the local command service, supply a strong bearer token, configure allowed Origins and start the server. It accepts authenticated POST requests to /v1/command on its chosen localhost port; a direct browser navigation or GET is not a valid command call. The daemon returns JSON for programmatic consumers, not a rendered migration dashboard.
Do not point production tools at this site's /api path or treat a JSON document as an installation wizard. The published CLI is the release-scoped local execution path.
Method: POST. Host: customer-controlled loopback IP. Path: /v1/command. Authorization: Bearer token, at least 32 bytes. Body: bounded JSON command envelope. Response: structured success or error JSON.
Supported source commands include inspect, plan, dry-run, approve, run, status, verify and receipt; check the exact released adapter before relying on additional commands. The website itself cannot connect to this bridge because its browser Content Security Policy disallows outbound requests.
Review the actual daemon contract → See installation and approval instructions →
Use Browser Studio when you need local CSV preparation and validation. Use the Local Runner when you need a production-verified, bounded-memory write into an existing SQLite table with a durable snapshot, approval, crash reconciliation and a receipt.
Profile, infer, transform, validate, checkpoint, inspect violations and export without a dataset backend. Browser input limit: 50 MiB.
Start in Studio →Bounded-memory stream + 256 MiB default source ceiling + durable snapshot + live target contract + target_write approval + fencing + atomic ledger + verification + receipt.
Open Local Runner →Clean a CSV in the browser, prepare a local SQLite migration, or connect an existing app. The supported boundary is different for each path.
# Preferred v1.1.1 install'));
+ assert.doesNotMatch(product, /GATE B · PRODUCTION-VERIFIED/);
+ assert.ok(product.includes('sustained-load qualification is still missing'));
+ assert.ok(product.includes('not a remotely hosted SPOOL API'));
+});
+
+test('developer links resolve to source documentation and mobile nav remains accessible', () => {
+ assert.ok(product.includes('href="${REPO}/blob/main/src/daemon/spoold.js"'));
+ assert.ok(!product.includes('\\' + '${REPO}'));
+ assert.match(styles, /\.start-paths/);
+ assert.match(styles, /\.developer-details/);
+ assert.match(styles, /\.guided-home-band/);
+ assert.match(styles, /\.product-topbar \.primary-nav\{display:flex/);
+});
From 37241a1cc204e72d9c4bba8d59c4e098ef59a8b6 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 16:52:07 +0530
Subject: [PATCH 02/21] fix(web): make all guided product routes first-class
SPA pages without transient 404
---
src/app.js | 28 ++++++-
src/product-surface.js | 21 ++++-
tests/product-route-bridge.test.js | 121 +++++++++++++++++++++++++++++
3 files changed, 165 insertions(+), 5 deletions(-)
create mode 100644 tests/product-route-bridge.test.js
diff --git a/src/app.js b/src/app.js
index 93a1408a..3491d45c 100644
--- a/src/app.js
+++ b/src/app.js
@@ -7,7 +7,8 @@ import { toolNamesForPhase } from './webmcp/registry.js';
import { initializeOptionalWebMcp } from './webmcp/optional.js';
const ROUTES = Object.freeze([
- '/', '/start', '/integrations', '/developers', '/autopilot', '/how-it-works', '/webmcp', '/benchmarks', '/docs',
+ '/', '/start', '/integrations', '/developers', '/local-runner', '/examples', '/security', '/services',
+ '/autopilot', '/how-it-works', '/webmcp', '/benchmarks', '/docs',
'/studio', '/studio/new', '/studio/mission', '/studio/results'
]);
@@ -430,6 +431,18 @@ function notFoundPage(path, state) {
function render() {
if (!latestState || !appRoot) return;
const path = currentPath();
+ const productMarkup = window.__spoolProductSurface?.render(path);
+ if (typeof productMarkup === 'string') {
+ appRoot.innerHTML = productMarkup;
+ document.title = `${window.__spoolProductSurface.title(path)} — SPOOL`;
+ return;
+ }
+ // The sibling product bundle might not have evaluated yet. Never paint
+ // an incorrect 404 while the valid product route initializes.
+ if (['/local-runner', '/examples', '/security', '/services', '/start', '/integrations', '/developers'].includes(path)) {
+ appRoot.innerHTML = 'Loading SPOOL page… ';
+ return;
+ }
const pages = {
'/': overviewPage,
'/autopilot': autopilotPage,
@@ -523,8 +536,17 @@ async function handleAction(action) {
function bindGlobalEvents() {
appRoot.addEventListener('click', event => {
- const routeLink = event.target.closest('[data-route]');
- if (routeLink) { event.preventDefault(); navigate(routeLink.dataset.route); return; }
+ // Product-page links and the Studio share one history without causing
+ // a new page load (or silently losing the browser-local workspace).
+ // Preserve native browser handling for modified clicks, downloads and
+ // links leaving this application's explicitly enumerated routes.
+ const routeLink = event.target.closest('[data-route], a[href^="/"]');
+ if (routeLink && !event.defaultPrevented && event.button === 0
+ && !event.metaKey && !event.ctrlKey && !event.shiftKey && !event.altKey
+ && routeLink.getAttribute('target') !== '_blank' && !routeLink.hasAttribute('download')) {
+ const route = routeLink.dataset.route ?? routeLink.getAttribute('href');
+ if (ROUTES.includes(route)) { event.preventDefault(); navigate(route); return; }
+ }
const outcome = event.target.closest('[data-outcome]');
if (outcome) { selectedOutcome = outcome.dataset.outcome; render(); return; }
const action = event.target.closest('[data-action]');
diff --git a/src/product-surface.js b/src/product-surface.js
index 5b73f99a..48713e7e 100644
--- a/src/product-surface.js
+++ b/src/product-surface.js
@@ -197,6 +197,23 @@ function productPage(path) {
})[path]?.();
}
+// The app and product modules are deliberately separate bundles. Export only
+// the rendering bridge needed by the existing SPA; do not expose a backend API
+// or mutable state in the browser global.
+const PRODUCT_TITLES = Object.freeze({
+ '/local-runner': 'Local Runner',
+ '/examples': 'Examples',
+ '/security': 'Security',
+ '/services': 'Services',
+ '/start': 'Start here',
+ '/integrations': 'Integrations',
+ '/developers': 'Developers'
+});
+window.__spoolProductSurface = Object.freeze({
+ render(path) { return PRODUCT_ROUTES.has(path) ? productPage(path) : null; },
+ title(path) { return PRODUCT_TITLES[path] ?? 'SPOOL'; }
+});
+
function addProductNav(root, path) {
const nav = root.querySelector('.primary-nav');
if (!nav) return;
@@ -219,7 +236,7 @@ function addHomepageProductBand(root) {
const section = document.createElement('section');
section.id = 'complete-product';
section.className = 'section complete-product-band';
- section.innerHTML = `THE COMPLETE PRODUCTTwo execution surfaces. One correctness model.
Use Browser Studio when you need local CSV preparation and validation. Use the Local Runner when you need a production-verified, bounded-memory write into an existing SQLite table with a durable snapshot, approval, crash reconciliation and a receipt.
BROWSER STUDIOWork entirely in-browser
Profile, infer, transform, validate, checkpoint, inspect violations and export without a dataset backend. Browser input limit: 50 MiB.
Start in Studio → LOCAL RUNNERWrite to a real SQLite target
Bounded-memory stream + 256 MiB default source ceiling + durable snapshot + live target contract + target_write approval + fencing + atomic ledger + verification + receipt.
Open Local Runner → `;
+ section.innerHTML = `THE COMPLETE PRODUCTTwo execution surfaces. One correctness model.
Use Browser Studio when you need local CSV preparation and validation. Use the Local Runner when you need a released, bounded-memory write into an existing SQLite table with a durable snapshot, approval, crash reconciliation and a receipt.
BROWSER STUDIOWork entirely in-browser
Profile, infer, transform, validate, checkpoint, inspect violations and export without a dataset backend. Browser input limit: 50 MiB.
Start in Studio → LOCAL RUNNERWrite to a real SQLite target
Bounded-memory stream + 256 MiB default source ceiling + durable snapshot + live target contract + target_write approval + fencing + atomic ledger + verification + receipt.
Open Local Runner → `;
finalCta.before(section);
const hero = root.querySelector('.hero');
if (hero && !root.querySelector('#choose-path')) {
@@ -239,7 +256,7 @@ function syncProductSurface() {
const current = root.querySelector(`[data-product-route="${path}"]`);
if (current) return;
root.innerHTML = productPage(path);
- document.title = `${({ '/local-runner': 'Local Runner', '/examples': 'Examples', '/security': 'Security', '/services': 'Services', '/start': 'Start here', '/integrations': 'Integrations', '/developers': 'Developers' })[path]} — SPOOL`;
+ document.title = `${PRODUCT_TITLES[path]} — SPOOL`;
return;
}
addProductNav(root, path);
diff --git a/tests/product-route-bridge.test.js b/tests/product-route-bridge.test.js
new file mode 100644
index 00000000..41db6584
--- /dev/null
+++ b/tests/product-route-bridge.test.js
@@ -0,0 +1,121 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+
+const appSource = readFileSync(new URL('../src/app.js', import.meta.url), 'utf8');
+const productSource = readFileSync(new URL('../src/product-surface.js', import.meta.url), 'utf8');
+const PRODUCT_PATHS = ['/local-runner', '/examples', '/security', '/services', '/start', '/integrations', '/developers'];
+
+function productBridge() {
+ const window = { location: { pathname: '/start' }, addEventListener() {} };
+ const document = { getElementById() { return null; } };
+ return new Function('window', 'document', 'MutationObserver', 'queueMicrotask',
+ productSource + '\nreturn window.__spoolProductSurface;'
+ )(window, document, class {}, () => {});
+}
+
+function appFunction(name, nextName, extra) {
+ const begin = appSource.indexOf('function ' + name + '(');
+ let end = appSource.indexOf('\nfunction ' + nextName + '(', begin);
+ if (end < 0) end = appSource.indexOf('\nasync function ' + nextName + '(', begin);
+ assert.ok(begin >= 0 && end > begin, name + ' must have a bounded definition');
+ return new Function(...extra.names, appSource.slice(begin, end) + '\nreturn ' + name + ';')(...extra.values);
+}
+
+test('every supported product page renders actual human-facing HTML without undefined placeholders', () => {
+ const bridge = productBridge();
+ for (const path of PRODUCT_PATHS) {
+ const html = bridge.render(path);
+ assert.ok(html.includes('data-product-route="' + path + '"'), path);
+ assert.ok(html.includes(' {
+ const bridge = productBridge();
+ for (const path of PRODUCT_PATHS) {
+ const root = { innerHTML: '' };
+ const document = { title: '' };
+ const render = appFunction('render', 'pageTitle', {
+ names: ['appRoot', 'latestState', 'currentPath', 'window', 'document'],
+ values: [root, {}, () => path, { __spoolProductSurface: bridge }, document]
+ });
+ render();
+ assert.equal(root.innerHTML, bridge.render(path), path);
+ assert.equal(document.title, bridge.title(path) + ' — SPOOL', path);
+ }
+});
+
+test('valid product URLs show loading state, never 404, if sibling bundle has not evaluated', () => {
+ const root = { innerHTML: '' };
+ const render = appFunction('render', 'pageTitle', {
+ names: ['appRoot', 'latestState', 'currentPath', 'window', 'document'],
+ values: [root, {}, () => '/start', {}, { title: '' }]
+ });
+ render();
+ assert.match(root.innerHTML, /Loading SPOOL/);
+ assert.doesNotMatch(root.innerHTML, /404/);
+});
+
+test('navigation accepts product URLs and rejects unknown routes', () => {
+ const routeDefinition = appSource.match(/const ROUTES = Object\.freeze\((\[[\s\S]*?\])\);/);
+ assert.ok(routeDefinition);
+ const routes = new Function('return ' + routeDefinition[1])();
+ for (const path of PRODUCT_PATHS) assert.ok(routes.includes(path), path);
+ const moved = [];
+ const window = {
+ location: { pathname: '/' },
+ history: { pushState(_, __, path) { moved.push(path); } },
+ scrollTo() {}
+ };
+ const navigate = appFunction('navigate', 'scheduleRender', {
+ names: ['ROUTES', 'window', 'render'],
+ values: [routes, window, () => {}]
+ });
+ navigate('/local-runner');
+ navigate('/start');
+ navigate('/not-a-route');
+ assert.deepEqual(moved, ['/local-runner', '/start']);
+});
+
+test('product-to-Studio clicks stay in SPA; modified, external, or download clicks stay native', () => {
+ let handler;
+ const routes = ['/', '/studio/new', '/start', '/local-runner'];
+ const moved = [];
+ const root = { addEventListener(type, listener) { if (type === 'click') handler = listener; } };
+ const window = { addEventListener() {} };
+ const bind = appFunction('bindGlobalEvents', 'initWebMcp', {
+ names: ['appRoot', 'window', 'ROUTES', 'navigate', 'render'],
+ values: [root, window, routes, route => moved.push(route), () => {}]
+ });
+ bind();
+ assert.equal(typeof handler, 'function');
+ function click(href, options = {}) {
+ let prevented = false;
+ const anchor = {
+ dataset: {},
+ getAttribute(name) { return name === 'href' ? href : null; },
+ hasAttribute(name) { return name === 'download' ? Boolean(options.download) : false; }
+ };
+ handler({
+ target: { closest(selector) { return selector === '[data-route], a[href^="/"]' ? anchor : null; } },
+ defaultPrevented: false,
+ button: 0,
+ ...options,
+ preventDefault() { prevented = true; }
+ });
+ return prevented;
+ }
+ assert.equal(click('/studio/new'), true);
+ assert.equal(click('/start'), true);
+ assert.equal(click('/start', { ctrlKey: true }), false);
+ assert.equal(click('/start', { metaKey: true }), false);
+ assert.equal(click('/start', { button: 1 }), false);
+ assert.equal(click('/start', { download: true }), false);
+ assert.equal(click('https://example.com/'), false);
+ assert.deepEqual(moved, ['/studio/new', '/start']);
+});
From fbb8b393324f8daaadb33288bce62919f2ae5971 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 16:54:26 +0530
Subject: [PATCH 03/21] fix(ci): refuse skipped or empty PostgreSQL suites on
standalone runner
---
docs/EXTERNAL_QUALIFICATION.md | 2 ++
scripts/qualify/assert-test-execution.js | 43 ++++++++++++++++++++++++
scripts/qualify/external-ci.sh | 6 ++--
tests/qualify-test-execution.test.js | 37 ++++++++++++++++++++
4 files changed, 86 insertions(+), 2 deletions(-)
create mode 100644 scripts/qualify/assert-test-execution.js
create mode 100644 tests/qualify-test-execution.test.js
diff --git a/docs/EXTERNAL_QUALIFICATION.md b/docs/EXTERNAL_QUALIFICATION.md
index 274395b9..1e821de3 100644
--- a/docs/EXTERNAL_QUALIFICATION.md
+++ b/docs/EXTERNAL_QUALIFICATION.md
@@ -40,3 +40,5 @@ For account-quota facts, consult the [official GitHub Actions billing documentat
## Qualification-soak duration is not optional
The named workloads are immutable inputs: CI = 100K rows / 30 seconds of CDC, SMALL = 1M / 1 hour, MEDIUM = 10M / 24 hours, LARGE = 100M / 72 hours, INDUSTRIAL = 1B / 7 days. The separate production qualification class also requires the complete cumulative identity, authority, crash, content-verification and HA gates in the readiness plan. Overrides receive a CUSTOM label and **never** inherit an immutable workload claim. The direct `external-ci.sh` command only executes the CI workload by default; its success cannot certify any higher class. Long-duration workloads require real isolated infrastructure and independent attestation; if no environment can execute them, the claim remains NOT_QUALIFIED.
+
+A test-process exit status alone is insufficient: standalone PostgreSQL conformance and red-team runners emit Node TAP and must have nonzero executed passing tests with exactly zero skips, TODO, failed or cancelled cases. The fail-closed `assert-test-execution.js` rejects empty, truncated or contradictory output. Both the transcripts and verification logs are checksummed in the external evidence bundle. This rule is mandatory even when PostgreSQL tests contain environment-based `skip` branches.
diff --git a/scripts/qualify/assert-test-execution.js b/scripts/qualify/assert-test-execution.js
new file mode 100644
index 00000000..f14daf61
--- /dev/null
+++ b/scripts/qualify/assert-test-execution.js
@@ -0,0 +1,43 @@
+#!/usr/bin/env node
+import { readFileSync } from 'node:fs';
+
+// An exit-code-zero TAP stream can contain ZERO executed tests, or only SKIPs.
+// Neither is a PostgreSQL qualification. The CI gate treats absent, malformed,
+// contradictory, omitted, or incomplete summaries as NOT_EXECUTED.
+export function assertExecutedTap(tap, { minimumTests = 1 } = {}) {
+ if (typeof tap !== 'string' || !Number.isSafeInteger(minimumTests) || minimumTests < 1) {
+ throw new TypeError('A TAP transcript and positive minimum test count are required');
+ }
+ const lines = tap.replace(/\r\n/g, '\n').split('\n');
+ if (!lines.some(line => /^TAP version 13\s*$/.test(line))) {
+ throw new Error('Node TAP v13 transcript header missing');
+ }
+ const fields = ['tests', 'pass', 'fail', 'cancelled', 'skipped', 'todo'];
+ const summary = {};
+ for (const name of fields) {
+ const entries = lines.filter(line => new RegExp('^# ' + name + ' [0-9]+\\s*$').test(line));
+ if (entries.length !== 1) throw new Error('TAP summary missing or ambiguous: ' + name);
+ summary[name] = Number(entries[0].match(/\d+/)[0]);
+ if (!Number.isSafeInteger(summary[name])) throw new Error('TAP counter not a safe integer: ' + name);
+ }
+ if (summary.tests < minimumTests) throw new Error('Insufficient executed tests: ' + summary.tests + ' < ' + minimumTests);
+ if (summary.pass !== summary.tests || summary.fail !== 0 || summary.cancelled !== 0
+ || summary.skipped !== 0 || summary.todo !== 0) {
+ throw new Error('TAP contains failed, cancelled, skipped, TODO, or unaccounted test cases');
+ }
+ if (!lines.some(line => /^ok \d+ - /.test(line))) throw new Error('No completed TAP test entries');
+ if (lines.some(line => /^not ok \d+ - /.test(line))) throw new Error('TAP reports failure despite clean summary');
+ return Object.freeze({ status: 'EXECUTED_ALL_PASS', ...summary });
+}
+
+if (process.argv[1] && /(?:^|[/\\])assert-test-execution\.js$/.test(process.argv[1])) {
+ try {
+ if (process.argv.length !== 4) throw new Error('Usage: node scripts/qualify/assert-test-execution.js FILE MINIMUM_TESTS');
+ const minimumTests = Number(process.argv[3]);
+ const result = assertExecutedTap(readFileSync(process.argv[2], 'utf8'), { minimumTests });
+ console.log(JSON.stringify(result));
+ } catch (error) {
+ console.error('POSTGRES_TEST_NOT_EXECUTED: ' + error.message);
+ process.exitCode = 1;
+ }
+}
diff --git a/scripts/qualify/external-ci.sh b/scripts/qualify/external-ci.sh
index 4f526b8c..401e2623 100644
--- a/scripts/qualify/external-ci.sh
+++ b/scripts/qualify/external-ci.sh
@@ -72,8 +72,10 @@ pg_tests=(tests/conformance/postgres*.test.js)
redteam_tests=(tests/industrial-postgres-*.test.js tests/industrial-redteam.test.js)
(( ${#pg_tests[@]} > 0 )) || fail 'no PostgreSQL conformance tests found'
(( ${#redteam_tests[@]} >= 2 )) || fail 'no complete PostgreSQL red-team tests found'
-run_gate postgres-conformance node --test --test-concurrency=1 "${pg_tests[@]}"
-run_gate postgres-redteam node --test --test-concurrency=1 "${redteam_tests[@]}"
+run_gate postgres-conformance node --test --test-reporter=tap --test-concurrency=1 "${pg_tests[@]}"
+run_gate postgres-conformance-executed node scripts/qualify/assert-test-execution.js "$evidence/postgres-conformance.log" 10
+run_gate postgres-redteam node --test --test-reporter=tap --test-concurrency=1 "${redteam_tests[@]}"
+run_gate postgres-redteam-executed node scripts/qualify/assert-test-execution.js "$evidence/postgres-redteam.log" 4
run_gate postgres-materialization node scripts/qualify-postgres-materialization.js
run_gate postgres-qualification node scripts/qualify/industrial-postgres.js --profile CI --output "$evidence/postgres"
run_gate evidence-verification node scripts/qualify/verify-evidence.js "$evidence/postgres"
diff --git a/tests/qualify-test-execution.test.js b/tests/qualify-test-execution.test.js
new file mode 100644
index 00000000..9d4d569c
--- /dev/null
+++ b/tests/qualify-test-execution.test.js
@@ -0,0 +1,37 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { assertExecutedTap } from '../scripts/qualify/assert-test-execution.js';
+
+const make = ({ tests = 2, pass = 2, fail = 0, cancelled = 0, skipped = 0, todo = 0, entries = ['ok 1 - first', 'ok 2 - second'] } = {}) =>
+ ['TAP version 13', ...entries, '# tests ' + tests, '# pass ' + pass, '# fail ' + fail,
+ '# cancelled ' + cancelled, '# skipped ' + skipped, '# todo ' + todo].join('\n');
+
+test('actual TAP with executed passing cases is accepted', () => {
+ const result = assertExecutedTap(make(), { minimumTests: 2 });
+ assert.equal(result.status, 'EXECUTED_ALL_PASS');
+ assert.equal(result.tests, 2);
+});
+test('a pass exit status with zero actual tests is rejected', () => {
+ assert.throws(() => assertExecutedTap(make({ tests: 0, pass: 0, entries: [] })), /Insufficient/);
+});
+test('all-skipped and partially skipped suites never count as qualified', () => {
+ for (const transcript of [
+ make({ pass: 0, skipped: 2, entries: ['ok 1 - no db # SKIP', 'ok 2 - no db # SKIP'] }),
+ make({ pass: 1, skipped: 1, entries: ['ok 1 - real', 'ok 2 - skip # SKIP'] })
+ ]) assert.throws(() => assertExecutedTap(transcript), /skipped/);
+});
+test('claimed PASS cannot override failed/cancelled/TODO or missing tests', () => {
+ for (const transcript of [
+ make({ pass: 1, fail: 1, entries: ['ok 1 - yes', 'not ok 2 - fail'] }),
+ make({ pass: 1, cancelled: 1 }),
+ make({ pass: 1, todo: 1 }),
+ make({ pass: 1 }),
+ make().replace('# pass 2', '')
+ ]) assert.throws(() => assertExecutedTap(transcript));
+});
+test('fake summaries with no test entries and partial transcripts are rejected', () => {
+ assert.throws(() => assertExecutedTap(make({ entries: [] })), /No completed/);
+ assert.throws(() => assertExecutedTap('TAP version 13\nok 1 - unresolved'), /summary/);
+ assert.throws(() => assertExecutedTap(make().replace('TAP version 13', '')), /header/);
+ assert.throws(() => assertExecutedTap(make() + '\n# pass 2'), /ambiguous/);
+});
From 9cf8fe00b64e87ab776c175d4d58bd14dcabb182 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 16:57:00 +0530
Subject: [PATCH 04/21] fix(qualify): make concurrent PostgreSQL CDC writer
mutations truly atomic
---
.../postgres/cdc-production-lifecycle.js | 26 ++++++--
tests/qualify-cdc-writer-transaction.test.js | 61 +++++++++++++++++++
2 files changed, 81 insertions(+), 6 deletions(-)
create mode 100644 tests/qualify-cdc-writer-transaction.test.js
diff --git a/scripts/qualify/postgres/cdc-production-lifecycle.js b/scripts/qualify/postgres/cdc-production-lifecycle.js
index ffd0ee85..5fddcd61 100644
--- a/scripts/qualify/postgres/cdc-production-lifecycle.js
+++ b/scripts/qualify/postgres/cdc-production-lifecycle.js
@@ -73,17 +73,27 @@ export async function runCdcProductionLifecycle({ profile, seed, evidenceWriter,
while (Date.now() < stopAt) {
const kind = (wi * 13 + opIndex * 7) % 100;
const baseId = rowCount + wi * 2_000_000 + opIndex;
- if (kind < 60) {
+ // A logical writer operation is exactly one source transaction. The
+ // delete+insert branch must not expose an intermediate deleted row to
+ // logical decoding. Do not claim "transactions" by counting loops.
+ await client.query('BEGIN');
+ try {
+ if (kind < 60) {
await client.query(`INSERT INTO public."${sourceTable}"(id, payload) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`, [baseId + 1, `w${wi}-${opIndex}`]);
- } else if (kind < 85) {
+ } else if (kind < 85) {
await client.query(`UPDATE public."${sourceTable}" SET payload = $2 WHERE id = $1`, [(opIndex % rowCount) + 1, `u${wi}-${opIndex}`]);
- } else {
+ } else {
await client.query(`DELETE FROM public."${sourceTable}" WHERE id = $1`, [(opIndex % rowCount) + 1]);
await client.query(`INSERT INTO public."${sourceTable}"(id, payload) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`, [(opIndex % rowCount) + 1, `r${wi}-${opIndex}`]);
+ }
+ await client.query('COMMIT');
+ transactions += 1;
+ } catch (error) {
+ try { await client.query('ROLLBACK'); } catch { /* original failure wins */ }
+ throw error;
}
ops += 1; opIndex += 1;
if (opIndex % 25 === 0) {
- transactions += 1;
await new Promise(resolve => setTimeout(resolve, 5));
}
}
@@ -170,7 +180,11 @@ export async function runCdcProductionLifecycle({ profile, seed, evidenceWriter,
const writerStats = await writerPromise;
const totalOps = writerStats.reduce((a, b) => a + b.ops, 0);
- evidenceWriter.appendRaw({ scenario, event: 'writers_quiesced', totalOps, writers });
+ const committedSourceTransactions = writerStats.reduce((a, b) => a + b.transactions, 0);
+ if (committedSourceTransactions !== totalOps) {
+ throw new Error(`Qualification writer transactions do not match committed operations: ${committedSourceTransactions}/${totalOps}`);
+ }
+ evidenceWriter.appendRaw({ scenario, event: 'writers_quiesced', totalOps, committedSourceTransactions, writers });
// 4. Catch up through the waterline on the production applier path.
const quencher = await admin();
@@ -240,7 +254,7 @@ export async function runCdcProductionLifecycle({ profile, seed, evidenceWriter,
scenario,
requestedConfig: { rows: profile.rows, writers: profile.writers, cdcDurationSeconds: profile.cdcDurationSeconds, largeTransactionRows: profile.largeTransactionRows },
effectiveConfig,
- rowCount, writers, durationSec, largeTxRows, totalOps,
+ rowCount, writers, durationSec, largeTxRows, totalOps, committedSourceTransactions,
appliedTransactions: summary.appliedTransactions ?? null,
appliedLsn: binding?.appliedLsn ?? null,
acknowledgedLsn: binding?.acknowledgedLsn ?? null,
diff --git a/tests/qualify-cdc-writer-transaction.test.js b/tests/qualify-cdc-writer-transaction.test.js
new file mode 100644
index 00000000..0b6d6bc3
--- /dev/null
+++ b/tests/qualify-cdc-writer-transaction.test.js
@@ -0,0 +1,61 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+
+const file = readFileSync(new URL('../scripts/qualify/postgres/cdc-production-lifecycle.js', import.meta.url), 'utf8');
+const start = file.indexOf(' async function writerTask(');
+const end = file.indexOf('\n try {\n const setup =', start);
+assert.ok(start >= 0 && end > start, 'executable qualification writer must exist');
+const writerBody = file.slice(start, end);
+
+async function executeOneWriterOperation({ worker = 0, failSql = null } = {}) {
+ const queries = [];
+ let closed = false;
+ let ticks = 0;
+ const client = {
+ async query(sql) {
+ queries.push(sql);
+ if (failSql && sql.includes(failSql)) throw new Error('injected source transaction fault');
+ return { rowCount: 1 };
+ },
+ async end() { closed = true; }
+ };
+ const fakeDate = { now() { return ticks++ === 0 ? 0 : 2; } };
+ const createWriter = new Function('admin', 'rowCount', 'sourceTable', 'Date', 'setTimeout',
+ writerBody + '\nreturn writerTask;'
+ );
+ const writer = createWriter(async () => client, 100, 'qualification_source', fakeDate,
+ () => Promise.resolve());
+ let result;
+ let error;
+ try { result = await writer(worker, 1); } catch (caught) { error = caught; }
+ return { queries, closed, result, error };
+}
+
+test('each seeded insert is committed as one actual PostgreSQL transaction', async () => {
+ const observed = await executeOneWriterOperation();
+ assert.deepEqual(observed.queries.map(query => query.split(' ')[0]),
+ ['BEGIN', 'INSERT', 'COMMIT']);
+ assert.deepEqual({ ops: observed.result.ops, transactions: observed.result.transactions },
+ { ops: 1, transactions: 1 });
+ assert.equal(observed.closed, true);
+});
+test('CDC delete-plus-reinsert is one atomic source transaction', async () => {
+ const observed = await executeOneWriterOperation({ worker: 7 }); // (7*13) % 100 = 91
+ assert.deepEqual(observed.queries.map(query => query.split(' ')[0]),
+ ['BEGIN', 'DELETE', 'INSERT', 'COMMIT']);
+ assert.equal(observed.result.transactions, 1);
+ assert.equal(observed.closed, true);
+});
+test('failure after delete rolls back both effects, never counts a committed transaction', async () => {
+ const observed = await executeOneWriterOperation({ worker: 7, failSql: 'INSERT' });
+ assert.deepEqual(observed.queries.map(query => query.split(' ')[0]),
+ ['BEGIN', 'DELETE', 'INSERT', 'ROLLBACK']);
+ assert.match(observed.error?.message ?? '', /injected source transaction fault/);
+ assert.equal(observed.closed, true);
+});
+test('qualification reports actual source commits, not loop-count approximations', () => {
+ assert.ok(file.includes('committedSourceTransactions !== totalOps'));
+ assert.ok(file.includes('event: \'writers_quiesced\', totalOps, committedSourceTransactions'));
+ assert.ok(!file.includes('transactions += 1;\n await new Promise'));
+});
From 90807ef923ea5cd32409dd340a0f4510b18d5852 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 17:03:42 +0530
Subject: [PATCH 05/21] fix(qualify): bound seven-day CDC soak and refuse
swallowed no-op driver failures
---
scripts/qualify/postgres/cdc-soak.js | 118 +++++++++++++++++----------
tests/qualify-cdc-soak.test.js | 87 ++++++++++++++++++++
2 files changed, 163 insertions(+), 42 deletions(-)
create mode 100644 tests/qualify-cdc-soak.test.js
diff --git a/scripts/qualify/postgres/cdc-soak.js b/scripts/qualify/postgres/cdc-soak.js
index b25abb91..e79c529e 100644
--- a/scripts/qualify/postgres/cdc-soak.js
+++ b/scripts/qualify/postgres/cdc-soak.js
@@ -62,30 +62,17 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
return res.rows[0].lsn;
};
- const CHUNKS = Math.max(4, Math.min(12, Math.floor(durationSec / 3)));
- const opsPerChunk = Math.max(4, Math.floor((durationSec * 1000) / (30 * CHUNKS)));
+ // Each 30-second slice allocates at most 100 operations per writer. A
+ // seven-day run cannot allocate billions of planned operations upfront.
+ const CHUNK_WINDOW_MS = 30_000;
+ const CHUNKS = Math.max(1, Math.ceil(durationSec * 1000 / CHUNK_WINDOW_MS));
+ const opsPerChunk = 100;
const writerPaceMs = Math.max(1, Math.floor((durationSec * 1000) / (CHUNKS * opsPerChunk)));
+ const WRITER_BASE = 100_000_000, WRITER_STRIDE = 20_000_000, MULTI_OFFSET = 10_000_000;
- // Deterministic per-chunk plan: op indices advance globally so capture order is stable.
- // update/delete ops may target rows that do not exist yet and emit no change message,
- // so equality is asserted on unique insert captures only; all captured changes still
- // carry monotonicity and resume-bound checks below.
- const plans = [];
+ // Only count SQL operations and inserts once the exact effect completed.
let expectedInsertCount = 0;
let eventsAttempted = 0;
- for (let chunk = 0; chunk < CHUNKS; chunk += 1) {
- const chunkPlans = Array.from({ length: writers }, () => []);
- for (let wi = 0; wi < writers; wi += 1) {
- for (let op = 0; op < opsPerChunk; op += 1) {
- const opIndex = chunk * opsPerChunk * writers + wi * opsPerChunk + op;
- const kind = opMix(wi, opIndex);
- eventsAttempted += opEvents(kind);
- if (kind === 'insert') expectedInsertCount += 1;
- chunkPlans[wi].push({ opIndex, kind });
- }
- }
- plans.push(chunkPlans);
- }
let capturedChanges = 0;
let insertCapturedTotal = 0;
@@ -97,34 +84,66 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
let windowBounded = true;
let lastLagBytes = null;
- async function writePlan(plan) {
- const tasks = Array.from({ length: writers }, async (_, wi) => {
- const wClient = new Client({ host: pgConfig.host, port: pgConfig.port, database: pgConfig.database, user: pgConfig.user, password: pgConfig.password, ssl: false });
- await wClient.connect();
+ async function writePlan(chunk) {
+ const jobs = Array.from({ length: writers }, async (_, wi) => {
+ const client = new Client({ host: pgConfig.host, port: pgConfig.port, database: pgConfig.database, user: pgConfig.user, password: pgConfig.password, ssl: false });
+ await client.connect();
+ let inserted = 0, attempted = 0, operations = 0;
try {
- const ops = plan[wi];
- for (let i = 0; i < ops.length; i += 1) {
- const { opIndex, kind } = ops[i];
- const baseId = wi * 1_000_000 + opIndex;
+ for (let op = 0; op < opsPerChunk; op += 1) {
+ const opIndex = chunk * opsPerChunk + op;
+ const kind = opMix(wi, opIndex);
+ const singleId = WRITER_BASE + wi * WRITER_STRIDE + opIndex + 1;
+ const id = kind === 'multi' ? singleId + MULTI_OFFSET : singleId;
+ if (opIndex >= MULTI_OFFSET || id > 2_147_483_647) {
+ throw new Error('CDC writer exhausted its disjoint 32-bit identifier band');
+ }
+ let inTx = false;
try {
- if (kind === 'insert') {
- await wClient.query(`INSERT INTO public."${table}"(id, payload) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`, [baseId + 1, `payload-${opIndex}`]);
- } else if (kind === 'update') {
- await wClient.query(`UPDATE public."${table}" SET payload = $2 WHERE id = $1`, [baseId % 200 + 1, `updated-${opIndex}`]);
+ if (kind !== 'insert') { await client.query('BEGIN'); inTx = true; }
+ const added = await client.query(`INSERT INTO public."${table}"(id, payload) VALUES ($1, $2)`, [id, `new-${wi}-${opIndex}`]);
+ if (added.rowCount !== 1) throw new Error('CDC writer insert was not exactly one row');
+ // All update/delete/multi events MUST mutate a row that exists.
+ // A setup INSERT and its mutation share one source transaction:
+ // this proves event order and atomicity rather than no-op SQL.
+ if (kind === 'update' || kind === 'multi') {
+ const updated = await client.query(`UPDATE public."${table}" SET payload = $2 WHERE id = $1`, [id, `updated-${wi}-${opIndex}`]);
+ if (updated.rowCount !== 1) throw new Error('CDC writer update was not exactly one row');
} else if (kind === 'delete') {
- await wClient.query(`DELETE FROM public."${table}" WHERE id = $1`, [baseId % 200 + 1]);
- } else {
- const multiId = baseId + 100_000;
- await wClient.query(`BEGIN; INSERT INTO public."${table}"(id, payload) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING; UPDATE public."${table}" SET payload = $3 WHERE id = $4; COMMIT`, [multiId, `m-${opIndex}`, `mu-${opIndex}`, multiId % 200 + 1]);
+ const deleted = await client.query(`DELETE FROM public."${table}" WHERE id = $1`, [id]);
+ if (deleted.rowCount !== 1) throw new Error('CDC writer delete was not exactly one row');
+ }
+ if (inTx) await client.query('COMMIT');
+ inserted += 1;
+ attempted += opEvents(kind);
+ operations += 1;
+ } catch (error) {
+ if (inTx) {
+ try { await client.query('ROLLBACK'); } catch { /* original failure wins */ }
}
- } catch {}
- if (i < ops.length - 1) await sleep(writerPaceMs);
+ throw error;
+ }
+ if (op < opsPerChunk - 1) await sleep(writerPaceMs);
}
} finally {
- await wClient.end();
+ await client.end();
}
+ return { inserted, attempted, operations };
});
- await Promise.all(tasks);
+ // Every writer must close before a failed run may drop the test tables.
+ const completed = await Promise.allSettled(jobs);
+ const rejected = completed.find(outcome => outcome.status === 'rejected');
+ if (rejected) throw rejected.reason;
+ const result = completed.reduce((acc, outcome) => {
+ acc.inserted += outcome.value.inserted;
+ acc.attempted += outcome.value.attempted;
+ acc.operations += outcome.value.operations;
+ return acc;
+ }, { inserted: 0, attempted: 0, operations: 0 });
+ if (result.operations !== writers * opsPerChunk) {
+ throw new Error('CDC workload did not execute every required writer operation');
+ }
+ return result;
}
// Streaming-ordering invariant: pgoutput emits complete transactions in commit
@@ -184,13 +203,17 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
// Steady-state capture: each chunk writes its deterministic plan, then the real
// SPOOL CDC connector streams from the durable cursor to the chunk waterline.
+ const soakStart = process.hrtime.bigint();
for (let chunk = 0; chunk < CHUNKS; chunk += 1) {
const chunkStart = Date.now();
- await writePlan(plans[chunk]);
+ const applied = await writePlan(chunk);
+ expectedInsertCount += applied.inserted;
+ eventsAttempted += applied.attempted;
// Flush anchor: a deterministic insert that is the last WAL record of the chunk,
// guaranteeing the streaming stop condition observes a commit at the waterline.
const anchorId = 5_000_000 + chunk;
- await adminClient.query(`INSERT INTO public."${table}"(id, payload) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`, [anchorId, `anchor-${chunk}`]);
+ const anchored = await adminClient.query(`INSERT INTO public."${table}"(id, payload) VALUES ($1, $2)`, [anchorId, `anchor-${chunk}`]);
+ if (anchored.rowCount !== 1) throw new Error('CDC anchor was not inserted exactly once');
expectedInsertCount += 1;
eventsAttempted += 1;
const untilLsn = await flushLsn(adminClient);
@@ -199,6 +222,11 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
startLsn: currentCheckpoint, untilLsn, maxMessages: 1_000_000
});
const rowChanges = streamed.changes.filter(change => ['insert', 'update', 'delete'].includes(change.tag) && change.table === table);
+ if (rowChanges.length !== applied.attempted + 1) {
+ failures.push({ code: 'CDC_ACTUAL_EVENT_COUNT_MISMATCH',
+ message: `chunk ${chunk} captured ${rowChanges.length} row events; expected ${applied.attempted + 1}`,
+ expected: false, classification: 'CORRECTNESS' });
+ }
capturedChanges += rowChanges.length;
insertCapturedTotal += rowChanges.filter(change => change.tag === 'insert').length;
checkResumeBound(rowChanges, currentCheckpoint);
@@ -211,6 +239,11 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
if (elapsed < writerPaceMs * opsPerChunk) await sleep(writerPaceMs * opsPerChunk - elapsed);
}
+ const observedSoakMs = Number(process.hrtime.bigint() - soakStart) / 1e6;
+ if (observedSoakMs < durationSec * 1000) {
+ failures.push({ code: 'CDC_SOAK_UNDER_EXECUTED', message: 'Monotonic CDC duration was shorter than declared workload', expected: false, classification: 'HARNESS' });
+ }
+
// Concurrent-window proof: a capture whose waterline was fixed BEFORE a burst
// commits must not see the burst (window bounded), and a fresh restart from that
// capture's durable cursor must absorb the burst exactly once (no loss, no replay).
@@ -320,6 +353,7 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
requestedConfig: { cdcDurationSeconds: profile.cdcDurationSeconds, writers: profile.writers },
effectiveConfig,
durationSec,
+ observedSoakMs,
writers,
chunks: CHUNKS,
eventsAttempted,
diff --git a/tests/qualify-cdc-soak.test.js b/tests/qualify-cdc-soak.test.js
new file mode 100644
index 00000000..9fec9d69
--- /dev/null
+++ b/tests/qualify-cdc-soak.test.js
@@ -0,0 +1,87 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+
+const code = readFileSync(new URL('../scripts/qualify/postgres/cdc-soak.js', import.meta.url), 'utf8');
+const start = code.indexOf(' async function writePlan(chunk) {');
+const end = code.indexOf(' // Streaming-ordering invariant:', start);
+assert.ok(start >= 0 && end > start, 'production CDC qualification writer is present');
+const functionBody = code.slice(start, end);
+
+async function exercise(kind, { workers = 1, fault = null, chunk = 0 } = {}) {
+ const sessions = [];
+ class Client {
+ constructor() { this.calls = []; this.closed = false; sessions.push(this); }
+ async connect() {}
+ async query(sql) {
+ this.calls.push(sql);
+ if (fault && sql.includes(fault)) throw new Error('injected PostgreSQL failure');
+ return { rowCount: 1 };
+ }
+ async end() { this.closed = true; }
+ }
+ const fn = new Function(
+ 'Client', 'pgConfig', 'writers', 'opsPerChunk', 'opMix', 'WRITER_BASE',
+ 'WRITER_STRIDE', 'MULTI_OFFSET', 'table', 'opEvents', 'sleep', 'writerPaceMs',
+ functionBody + '\nreturn writePlan;'
+ )(
+ Client, { host: '127.0.0.1', port: 5432, database: 'isolated', user: 'test', password: 'fixture' },
+ workers, 100, () => kind, 100_000_000, 20_000_000, 10_000_000,
+ 'q_cdc_test', value => value === 'insert' ? 1 : 2, () => Promise.resolve(), 0
+ );
+ let result, error;
+ try { result = await fn(chunk); } catch (caught) { error = caught; }
+ return { result, error, sessions };
+}
+
+test('seven-day CDC workload is generated in bounded windows, not preallocated arrays', () => {
+ assert.match(code, /const CHUNK_WINDOW_MS = 30_000/);
+ assert.match(code, /const opsPerChunk = 100/);
+ assert.doesNotMatch(code, /const plans =|plans\[chunk\]/);
+ assert.match(code, /const observedSoakMs = Number\(process\.hrtime\.bigint\(\) - soakStart\)/);
+ assert.match(code, /CDC_SOAK_UNDER_EXECUTED/);
+ const chunks = Math.ceil(7 * 86400 * 1000 / 30000);
+ assert.equal(chunks, 20160);
+ const maxOpIndex = chunks * 100 - 1;
+ const lastId = 100_000_000 + 49 * 20_000_000 + maxOpIndex + 1 + 10_000_000;
+ assert.ok(lastId < 2_147_483_647);
+});
+
+for (const kind of ['insert', 'update', 'delete', 'multi']) {
+ test('real source mutations and counted CDC effects: ' + kind, async () => {
+ const { result, error, sessions } = await exercise(kind);
+ assert.equal(error, undefined);
+ assert.equal(result.operations, 100);
+ assert.equal(result.inserted, 100);
+ assert.equal(result.attempted, kind === 'insert' ? 100 : 200);
+ assert.equal(sessions.length, 1);
+ assert.equal(sessions[0].closed, true);
+ const log = sessions[0].calls.map(query => query.split(' ')[0]);
+ if (kind === 'insert') assert.deepEqual(log.slice(0, 1), ['INSERT']);
+ else assert.deepEqual(log.slice(0, 4), ['BEGIN', 'INSERT', kind === 'delete' ? 'DELETE' : 'UPDATE', 'COMMIT']);
+ assert.equal(log.some(sql => sql.includes(';')), false, 'node-postgres parameterized multi-statements forbidden');
+ });
+}
+
+test('concurrent fault rolls back incomplete transaction and awaits every worker shutdown', async () => {
+ const { result, error, sessions } = await exercise('multi', { workers: 2, fault: 'UPDATE' });
+ assert.equal(result, undefined);
+ assert.match(error?.message ?? '', /injected PostgreSQL failure/);
+ assert.equal(sessions.length, 2);
+ assert.ok(sessions.every(client => client.closed));
+ assert.ok(sessions.every(client => client.calls.includes('ROLLBACK')));
+ assert.ok(sessions.every(client => !client.calls.includes('COMMIT')));
+});
+
+test('identifier-space exhaustion fails instead of aliasing another writer or marker', async () => {
+ const { result, error, sessions } = await exercise('insert', { chunk: 100000 });
+ assert.equal(result, undefined);
+ assert.match(error?.message ?? '', /exhausted/);
+ assert.ok(sessions.every(client => client.closed));
+});
+
+test('a driver exception cannot be silently counted as completed qualification evidence', () => {
+ assert.doesNotMatch(functionBody, /catch\s*\{\s*\}/);
+ assert.match(functionBody, /if \(insertedRow\.rowCount|if \(added\.rowCount/);
+ assert.match(code, /CDC_ACTUAL_EVENT_COUNT_MISMATCH/);
+});
From 50055702ce79d89e48bbe546b14078afe48da8ef Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 17:06:10 +0530
Subject: [PATCH 06/21] fix(qualify): test genuine exported MVCC snapshot
across all declared reader sessions
---
.../qualify/postgres/database-source-load.js | 89 +++++++++++------
tests/qualify-snapshot-fanout.test.js | 96 +++++++++++++++++++
2 files changed, 154 insertions(+), 31 deletions(-)
create mode 100644 tests/qualify-snapshot-fanout.test.js
diff --git a/scripts/qualify/postgres/database-source-load.js b/scripts/qualify/postgres/database-source-load.js
index 0de7906c..6c271ff5 100644
--- a/scripts/qualify/postgres/database-source-load.js
+++ b/scripts/qualify/postgres/database-source-load.js
@@ -37,40 +37,67 @@ export async function runDatabaseSourceLoad({ profile, seed, evidenceWriter, pgC
const catalog = await discoverPostgresCatalog({ ref, credentialBroker: broker, schemas: ['public'] });
evidenceWriter.appendRaw({ scenario, event: 'catalog_discovered', tableCount: catalog.tableCount, catalogId: catalog.catalogId });
- // Export snapshot AND read inside the same owning transaction.
- // The exported token is only joinable while the exporter holds its tx open,
- // so all consistency reads must happen inside the callback.
- const snapshotResult = await withExportedPostgresSnapshot({ ref, credentialBroker: broker }, async (snapshot) => {
- evidenceWriter.appendRaw({ scenario, event: 'snapshot_exported', exportedSnapshotId: snapshot.exportedSnapshotId, walLsn: snapshot.walLsn });
- // Read two tables via the shared MVCC snapshot by joining inside separate
- // sessions while the exporter is still open (exporter commits on return).
- // NOTE: joinPostgresSnapshot opens its own BEGIN; the token is valid here.
+ // The exporter must remain open while all child sessions join its MVCC
+ // snapshot. A write performed AFTER the snapshot closes proves nothing.
+ const snapshotResult = await withExportedPostgresSnapshot({ ref, credentialBroker: broker }, async snapshot => {
+ evidenceWriter.appendRaw({ scenario, event: 'snapshot_exported',
+ exportedSnapshotId: snapshot.exportedSnapshotId, walLsn: snapshot.walLsn });
+
+ // An INDEPENDENT admin session commits post-export mutations BEFORE
+ // any child reader joins. Every live table now has 1001 rows while
+ // every reader joined to the old snapshot must still see 1000.
+ for (const t of tables) {
+ await admin.query(`INSERT INTO public."${t}"(id, payload) VALUES (1000001, 'post-snapshot')`);
+ }
+
+ // Exactly profile.writers concurrent sessions, each joining two tables.
+ // The old harness created 2*writers tables but silently read only two.
const runtime = createPostgresSourceRuntime({ credentialBroker: broker, batchSize: 1000 });
- let totalRows = 0;
- const readTables = tables.slice(0, 2);
- for (const t of readTables) {
- const sourceRef = { connector: 'postgres', endpoint: `postgresql://${pgConfig.host}:${pgConfig.port}?sslmode=disable`, database: pgConfig.database, schema: 'public', table: t, identity: { user: pgConfig.user }, secretRef: { provider: 'env', key: 'PGPASSWORD' } };
- const count = await runtime.withSnapshot(sourceRef, async (s) => {
- let c = 0;
- for await (const b of s.batches()) c += b.rows.length;
- return c;
- }, { snapshotToken: snapshot.exportedSnapshotId });
- totalRows += count;
+ const readerStats = await Promise.all(Array.from({ length: profile.writers }, async (_, wi) => {
+ let readRows = 0;
+ for (const t of tables.slice(wi * 2, wi * 2 + 2)) {
+ const sourceRef = {
+ connector: 'postgres', endpoint: ref.endpoint, database: pgConfig.database,
+ schema: 'public', table: t, identity: { user: pgConfig.user },
+ secretRef: { provider: 'env', key: 'PGPASSWORD' }
+ };
+ const count = await runtime.withSnapshot(sourceRef, async joined => {
+ let c = 0;
+ for await (const batch of joined.batches()) c += batch.rows.length;
+ return c;
+ }, { snapshotToken: snapshot.exportedSnapshotId });
+ if (count !== 1000) {
+ throw new Error(`Joined reader ${wi} observed ${count} rows in ${t}; exported snapshot should contain 1000`);
+ }
+ readRows += count;
+ }
+ return { writer: wi, rows: readRows, tablesRead: 2 };
+ }));
+ const totalRows = readerStats.reduce((total, reader) => total + reader.rows, 0);
+ if (readerStats.length !== profile.writers || totalRows !== tableCount * 1000) {
+ throw new Error('Exported-snapshot fanout did not execute its full declared reader count');
}
- // Concurrent-write check: insert AFTER snapshot export; those rows must
- // NOT be visible to the joined readers above (they already completed) —
- // we verify by re-reading outside the snapshot and confirming divergence.
- return { exportedSnapshotId: snapshot.exportedSnapshotId, walLsn: snapshot.walLsn, catalogId: catalog.catalogId, totalRows };
- });
- evidenceWriter.appendRaw({ scenario, event: 'database_source_completed', totalRows: snapshotResult.totalRows, tableCount });
+ // Prove the writes were real and visible outside the exported snapshot,
+ // while all joined readers above saw only the pre-export rows.
+ for (const t of tables) {
+ const live = await admin.query(`SELECT count(*)::int AS c FROM public."${t}"`);
+ if (live.rows[0]?.c !== 1001) {
+ throw new Error(`Post-export write was not visible to an independent session for ${t}`);
+ }
+ }
+ return {
+ exportedSnapshotId: snapshot.exportedSnapshotId,
+ walLsn: snapshot.walLsn, catalogId: catalog.catalogId, totalRows,
+ joinedReaders: readerStats.length,
+ tablesRead: readerStats.reduce((total, reader) => total + reader.tablesRead, 0),
+ postExportWrites: tables.length
+ };
+ });
- // Post-snapshot write: prove the snapshot was point-in-time by showing the
- // live table now has rows the snapshot read did not see.
- await admin.query(`INSERT INTO public."${tables[0]}"(id, payload) VALUES (1000001, 'post-snapshot')`);
- const liveCount = await admin.query(`SELECT count(*)::int as c FROM public."${tables[0]}"`);
- if (liveCount.rows[0].c !== 1001) throw new Error(`expected 1001 live rows after post-snapshot write, got ${liveCount.rows[0].c}`);
- if (snapshotResult.totalRows !== 2000) throw new Error(`expected snapshot readers to see 2000 rows (2x1000), saw ${snapshotResult.totalRows}`);
+ evidenceWriter.appendRaw({ scenario, event: 'database_source_completed',
+ totalRows: snapshotResult.totalRows, tableCount, joinedReaders: snapshotResult.joinedReaders,
+ tablesRead: snapshotResult.tablesRead, postExportWrites: snapshotResult.postExportWrites });
await admin.end(); admin = null;
@@ -79,7 +106,7 @@ export async function runDatabaseSourceLoad({ profile, seed, evidenceWriter, pgC
for (const t of tables) await cleanup.query(`DROP TABLE IF EXISTS public."${t}" CASCADE`);
await cleanup.end();
- return { metrics: { scenario, requestedConfig: { writers: profile.writers }, effectiveConfig, tableCount, totalRows: snapshotResult.totalRows, catalogId: catalog.catalogId, snapshotId: snapshotResult.exportedSnapshotId }, failures: [] };
+ return { metrics: { scenario, requestedConfig: { writers: profile.writers }, effectiveConfig, tableCount, totalRows: snapshotResult.totalRows, joinedReaders: snapshotResult.joinedReaders, tablesRead: snapshotResult.tablesRead, postExportWrites: snapshotResult.postExportWrites, catalogId: catalog.catalogId, snapshotId: snapshotResult.exportedSnapshotId }, failures: [] };
} finally {
try { if (admin) await admin.end(); } catch {}
try {
diff --git a/tests/qualify-snapshot-fanout.test.js b/tests/qualify-snapshot-fanout.test.js
new file mode 100644
index 00000000..5ffe6dd1
--- /dev/null
+++ b/tests/qualify-snapshot-fanout.test.js
@@ -0,0 +1,96 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+
+const source = readFileSync(new URL('../scripts/qualify/postgres/database-source-load.js', import.meta.url), 'utf8');
+
+async function qualifyMock(writers, { wrongJoined = false, missingLive = false } = {}) {
+ const tables = new Map(), writes = [], readers = [];
+ let exporterOpen = false;
+ class Client {
+ async connect() {}
+ async end() {}
+ async query(sql) {
+ const table = /public\."([^"]+)"/.exec(sql)?.[1];
+ if (sql.startsWith('DROP TABLE')) { tables.delete(table); return { rowCount: 0 }; }
+ if (sql.startsWith('CREATE TABLE')) { tables.set(table, 0); return { rowCount: 0 }; }
+ if (sql.includes('generate_series(1, 1000)')) { tables.set(table, 1000); return { rowCount: 1000 }; }
+ if (sql.includes('VALUES (1000001')) {
+ assert.equal(exporterOpen, true, 'post-export write happened after snapshot owner closed');
+ tables.set(table, tables.get(table) + 1);
+ writes.push(table);
+ return { rowCount: 1 };
+ }
+ if (sql.startsWith('SELECT count(*)::int')) {
+ return { rows: [{ c: missingLive && table === writes[0] ? 1000 : tables.get(table) }] };
+ }
+ throw new Error('Unexpected mock SQL: ' + sql);
+ }
+ }
+ const fakeHash = () => ({ update() { return this; }, digest() { return 'abcdef0123456789'; } });
+ const sourceRuntime = () => ({
+ async withSnapshot(ref, read, opts) {
+ assert.equal(exporterOpen, true, 'joined reader ran after export transaction ended');
+ assert.equal(writes.length, writers * 2, 'read began before all post-export writes');
+ assert.equal(opts.snapshotToken, 'snapshot-1');
+ readers.push(ref.table);
+ const count = wrongJoined && readers.length === 1 ? 1001 : 1000;
+ return read({ async *batches() { yield { rows: Array(count) }; } });
+ }
+ });
+ const executable = source.replace(/^import[\s\S]*?;\s*$/gm, '').replace(/^export /gm, '');
+ const qualify = new Function(
+ 'createHash', 'Client', 'CredentialBroker', 'discoverPostgresCatalog',
+ 'withExportedPostgresSnapshot', 'createPostgresSourceRuntime',
+ executable + '\nreturn runDatabaseSourceLoad;'
+ )(
+ fakeHash, Client, class {},
+ async () => ({ tableCount: tables.size, catalogId: 'catalog-1' }),
+ async (_, callback) => {
+ exporterOpen = true;
+ try { return await callback({ exportedSnapshotId: 'snapshot-1', walLsn: '0/A' }); }
+ finally { exporterOpen = false; }
+ },
+ sourceRuntime
+ );
+ let result, error;
+ try {
+ result = await qualify({
+ profile: { writers }, seed: 'fixture', runId: 'fixture',
+ evidenceWriter: { appendRaw() {} },
+ pgConfig: { host: '127.0.0.1', port: 5432, database: 'isolated', user: 'test', password: 'test-only' }
+ });
+ } catch (caught) { error = caught; }
+ return { result, error, writes, readers, remainingTables: tables.size };
+}
+
+test('all declared source readers join the exported snapshot, not two hardcoded tables', async () => {
+ for (const writers of [2, 5]) {
+ const { result, error, writes, readers, remainingTables } = await qualifyMock(writers);
+ assert.equal(error, undefined);
+ assert.equal(writes.length, writers * 2);
+ assert.equal(new Set(readers).size, writers * 2);
+ assert.equal(result.metrics.joinedReaders, writers);
+ assert.equal(result.metrics.tablesRead, writers * 2);
+ assert.equal(result.metrics.postExportWrites, writers * 2);
+ assert.equal(result.metrics.totalRows, writers * 2000);
+ assert.equal(remainingTables, 0);
+ }
+});
+test('reader seeing post-export rows fails the shared-MVCC-snapshot qualification', async () => {
+ const result = await qualifyMock(2, { wrongJoined: true });
+ assert.match(result.error?.message ?? '', /Joined reader/);
+ assert.equal(result.result, undefined);
+ assert.equal(result.remainingTables, 0);
+});
+test('post-export write must be independently visible outside joined snapshot', async () => {
+ const result = await qualifyMock(2, { missingLive: true });
+ assert.match(result.error?.message ?? '', /not visible/);
+ assert.equal(result.result, undefined);
+ assert.equal(result.remainingTables, 0);
+});
+test('source code no longer substitutes a post-closure write for a concurrent snapshot test', () => {
+ assert.doesNotMatch(source, /tables\.slice\(0, 2\)/);
+ assert.match(source, /await Promise\.all\(Array\.from\(\{ length: profile\.writers \}/);
+ assert.match(source, /postExportWrites: snapshotResult\.postExportWrites/);
+});
From cf83f291dea95b5a29eb6197a83220501a0da040 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 18:26:54 +0530
Subject: [PATCH 07/21] fix(cdc): serialize bounded pgoutput callbacks and
acknowledge only durable complete events
---
src/connectors/postgres/cdc-serial-pump.js | 37 ++++++
src/connectors/postgres/cdc.js | 55 ++++++---
tests/postgres-cdc-capture-ordering.test.js | 130 ++++++++++++++++++++
3 files changed, 204 insertions(+), 18 deletions(-)
create mode 100644 src/connectors/postgres/cdc-serial-pump.js
create mode 100644 tests/postgres-cdc-capture-ordering.test.js
diff --git a/src/connectors/postgres/cdc-serial-pump.js b/src/connectors/postgres/cdc-serial-pump.js
new file mode 100644
index 00000000..02fe65bf
--- /dev/null
+++ b/src/connectors/postgres/cdc-serial-pump.js
@@ -0,0 +1,37 @@
+// EventEmitter ignores promises from async listeners. Serialize WAL callbacks
+// so callback validation/durability and acknowledgement stay in source order.
+export function createCdcSerialPump({ handle, onFailure, maxPending = 1024 } = {}) {
+ if (typeof handle !== 'function' || typeof onFailure !== 'function') {
+ throw new TypeError('CDC serial pump requires handler and failure callback');
+ }
+ if (!Number.isSafeInteger(maxPending) || maxPending < 1 || maxPending > 100_000) {
+ throw new TypeError('CDC maxPending must be 1..100000');
+ }
+ let accepting = true;
+ let pending = 0;
+ let serial = Promise.resolve();
+ function stopWith(error) {
+ if (!accepting) return;
+ accepting = false;
+ try { onFailure(error); } catch { /* original failure wins */ }
+ }
+ return Object.freeze({
+ enqueue(lsn, message) {
+ if (!accepting) return false;
+ if (pending >= maxPending) {
+ stopWith(Object.assign(new Error('CDC callback backlog exceeded the configured bound'), {
+ code: 'CDC_EVENT_BACKPRESSURE_EXCEEDED'
+ }));
+ return false;
+ }
+ pending += 1;
+ serial = serial.then(async () => {
+ if (accepting) await handle(lsn, message);
+ }).catch(stopWith).finally(() => { pending -= 1; });
+ return true;
+ },
+ close() { accepting = false; },
+ drain() { return serial; },
+ stats() { return Object.freeze({ accepting, pending }); }
+ });
+}
diff --git a/src/connectors/postgres/cdc.js b/src/connectors/postgres/cdc.js
index c4c56b20..44803041 100644
--- a/src/connectors/postgres/cdc.js
+++ b/src/connectors/postgres/cdc.js
@@ -3,6 +3,7 @@ import { resolve } from 'node:path';
import { Client } from 'pg';
import { LogicalReplicationService, PgoutputPlugin } from 'pg-logical-replication';
import { fail } from '../../core/errors.js';
+import { createCdcSerialPump } from './cdc-serial-pump.js';
import { expressionSourceFields } from '../../core/transforms.js';
import { sha256Canonical } from '../../platform/canonical-json.js';
import { parsePostgresEndpoint, postgresConnectionConfig } from './connection.js';
@@ -597,37 +598,53 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
resolveDone();
};
- service.on('error', error => finish(error));
- service.on('data', async (lsn, message) => {
- if (finished) return;
- try {
- const change = normalizeMessage(typeof lsn === 'string' ? lsn : String(lsn ?? ''), message);
+ // WAL callbacks may be emitted while a previous async onChange is
+ // still running. Serial processing prevents early/out-of-order ACKs.
+ const pump = createCdcSerialPump({
+ maxPending: 1024,
+ onFailure(error) {
+ finish(error);
+ void service.stop().catch(() => { /* first failure wins */ });
+ },
+ async handle(lsn, message) {
+ if (finished) return;
+ const position = typeof lsn === 'string' ? lsn : String(lsn ?? '');
+ const change = normalizeMessage(position, message);
+ if (change && maxMessages != null && messageCount + 1 >= maxMessages && change.tag !== 'commit') {
+ fail('CDC_CAPTURE_BOUNDARY_INCOMPLETE', 'Maximum CDC capture messages reached before a committed transaction boundary');
+ }
if (change) {
- if (onChange) await onChange(change);
- // Checkpoint monotonicity: commit LSN must not go backwards (per pgoutput commit-order guarantee)
if (change.tag === 'commit' && lastAcknowledgedLsn && comparePostgresLsn(change.lsn, lastAcknowledgedLsn) < 0) {
- throw fail('CDC_LSN_MONOTONICITY_VIOLATION', `Commit LSN went backwards: ${lastAcknowledgedLsn} -> ${change.lsn}`);
- }
- if (change.tag === 'commit') {
- lastAcknowledgedLsn = change.lsn;
+ fail('CDC_LSN_MONOTONICITY_VIOLATION', `Commit LSN went backwards: ${lastAcknowledgedLsn} -> ${change.lsn}`);
}
+ if (onChange) await onChange(change);
+ }
+ // Never advance the reported acknowledged LSN before the transport
+ // confirms acknowledgement; no ACK when onChange rejects.
+ await service.acknowledge(position);
+ if (change) {
+ if (change.tag === 'commit') lastAcknowledgedLsn = change.lsn;
messageCount += 1;
- // Bounded changes array for testing - drop oldest if over limit
if (changes.length >= MAX_CHANGES) changes.shift();
changes.push(change);
}
- await service.acknowledge(typeof lsn === 'string' ? lsn : String(lsn ?? ''));
+ // Never present a partial source transaction as a successful bounded
+ // capture. A maximum-message breach mid-transaction is an explicit
+ // refusal; LSN waterlines only end at a complete COMMIT.
const stopByCount = maxMessages != null && messageCount >= maxMessages;
- const stopByLsn = untilLsn != null && change && comparePostgresLsn(change.lsn, untilLsn) >= 0;
+ const stopByLsn = untilLsn != null && change?.tag === 'commit'
+ && comparePostgresLsn(change.lsn, untilLsn) >= 0;
if (stopByCount || stopByLsn) {
- try { await service.stop(); } catch { /* stopping wins over stop errors */ }
+ await service.stop();
finish(null);
}
- } catch (error) {
- try { await service.stop(); } catch { /* stopping wins over stop errors */ }
- finish(error);
}
});
+ service.on('error', error => {
+ pump.close();
+ finish(error);
+ });
+ service.on('data', (lsn, message) => { pump.enqueue(lsn, message); });
try {
await service.subscribe(new PgoutputPlugin({ protoVersion: 1, publicationNames: [...publicationNames] }), name, startLsn ?? undefined);
@@ -636,6 +653,8 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
finish(error);
await done;
} finally {
+ pump.close();
+ await pump.drain();
try { await service.destroy(); } catch { /* shutdown must not mask results */ }
}
// Translate transport failures into operator-actionable CDC errors at the
diff --git a/tests/postgres-cdc-capture-ordering.test.js b/tests/postgres-cdc-capture-ordering.test.js
new file mode 100644
index 00000000..7863b92b
--- /dev/null
+++ b/tests/postgres-cdc-capture-ordering.test.js
@@ -0,0 +1,130 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+import { createCdcSerialPump } from '../src/connectors/postgres/cdc-serial-pump.js';
+
+const code = readFileSync(new URL('../src/connectors/postgres/cdc.js', import.meta.url), 'utf8');
+const begin = code.indexOf('export async function streamPostgresChanges(');
+const end = code.indexOf('export function createCdcCheckpointStore(', begin);
+assert.ok(begin >= 0 && end > begin, 'real CDC capture implementation is present');
+const captureSource = code.slice(begin, end).replace(/^export /gm, '');
+
+function makeCapture(events, { failAcknowledging = null } = {}) {
+ let service;
+ class FakeReplicationService {
+ constructor() {
+ this.listeners = new Map(); this.acknowledged = [];
+ this.stopped = false; this.destroyed = false; service = this;
+ }
+ on(name, cb) { this.listeners.set(name, cb); }
+ async subscribe() {
+ for (const [lsn, kind] of events) this.listeners.get('data')(lsn, { tag: kind });
+ }
+ async acknowledge(lsn) {
+ if (lsn === failAcknowledging) throw Object.assign(new Error('injected ACK failure'), { code: 'ACK_FAILED' });
+ this.acknowledged.push(lsn);
+ }
+ async stop() { this.stopped = true; }
+ async destroy() { this.destroyed = true; }
+ }
+ const fail = (code, message) => { throw Object.assign(new Error(message), { code }); };
+ const capture = new Function(
+ 'requireDatabaseRef', 'requireSlotName', 'PUBLICATION_NAME', 'LSN',
+ 'parsePostgresEndpoint', 'LogicalReplicationService', 'PgoutputPlugin',
+ 'normalizeMessage', 'comparePostgresLsn', 'fail',
+ 'classifyCdcStreamError', 'createCdcSerialPump',
+ captureSource + '\nreturn streamPostgresChanges;'
+ )(
+ ref => ref, slot => slot, /^[a-z_]+$/, /^0\/[0-9A-Fa-f]+$/,
+ () => ({ host: 'localhost', port: 5432, sslMode: 'disable' }),
+ FakeReplicationService, class {},
+ (lsn, message) => ({ tag: message.tag, lsn }),
+ (left, right) => Number.parseInt(left.split('/')[1], 16) - Number.parseInt(right.split('/')[1], 16),
+ fail, error => { throw error; }, createCdcSerialPump
+ );
+ const run = extra => capture({
+ ref: { endpoint: 'postgresql://127.0.0.1:5432?sslmode=disable', database: 'isolated', identity: { user: 'test' }, secretRef: { provider: 'env', key: 'ISOLATED_TEST' } },
+ credentialBroker: { withSecret: async (_, fn) => fn('fake') },
+ slotName: 'isolated_slot', publicationNames: ['isolated_pub'],
+ startLsn: '0/0', untilLsn: '0/3', ...extra
+ });
+ return { run, getService: () => service };
+}
+
+const events = [['0/1', 'begin'], ['0/2', 'insert'], ['0/3', 'commit']];
+
+test('slow durability callback cannot cause a later message to acknowledge first', async () => {
+ let resume;
+ const gate = new Promise(resolve => { resume = resolve; });
+ const harness = makeCapture(events);
+ const observed = [];
+ const running = harness.run({
+ onChange: async change => {
+ if (change.tag === 'begin') await gate;
+ observed.push(change.tag);
+ }
+ });
+ for (let i = 0; i < 5; i++) await Promise.resolve();
+ assert.deepEqual(harness.getService().acknowledged, []);
+ resume();
+ const result = await running;
+ assert.deepEqual(observed, ['begin', 'insert', 'commit']);
+ assert.deepEqual(harness.getService().acknowledged, ['0/1', '0/2', '0/3']);
+ assert.equal(result.acknowledgedLsn, '0/3');
+ assert.equal(harness.getService().destroyed, true);
+});
+
+test('durability failure refuses subsequent messages and never ACKs their WAL positions', async () => {
+ const harness = makeCapture(events);
+ await assert.rejects(harness.run({
+ onChange: async change => {
+ if (change.tag === 'insert') throw Object.assign(new Error('injected apply failure'), { code: 'INJECTED_FAILURE' });
+ }
+ }), { code: 'INJECTED_FAILURE' });
+ assert.deepEqual(harness.getService().acknowledged, ['0/1']);
+ assert.equal(harness.getService().destroyed, true);
+});
+
+test('maximum message limit refuses incomplete source transaction before ACKing the boundary event', async () => {
+ const harness = makeCapture(events);
+ const delivered = [];
+ await assert.rejects(harness.run({
+ maxMessages: 2, onChange: async change => { delivered.push(change.tag); }
+ }), { code: 'CDC_CAPTURE_BOUNDARY_INCOMPLETE' });
+ assert.deepEqual(delivered, ['begin']);
+ assert.deepEqual(harness.getService().acknowledged, ['0/1']);
+});
+
+test('LSN waterline is applied only on COMMIT, not to uncommitted row messages', async () => {
+ const harness = makeCapture(events);
+ const result = await harness.run({ untilLsn: '0/2' });
+ assert.deepEqual(harness.getService().acknowledged, ['0/1', '0/2', '0/3']);
+ assert.equal(result.acknowledgedLsn, '0/3');
+});
+
+test('transport ACK rejection cannot advance the reported commit LSN', async () => {
+ const harness = makeCapture(events, { failAcknowledging: '0/3' });
+ await assert.rejects(harness.run(), { code: 'ACK_FAILED' });
+ assert.deepEqual(harness.getService().acknowledged, ['0/1', '0/2']);
+});
+
+test('callback queue bound refuses overflow without processing subsequent events', async () => {
+ let release;
+ const gate = new Promise(resolve => { release = resolve; });
+ const applied = [];
+ const errors = [];
+ const pump = createCdcSerialPump({
+ maxPending: 2,
+ handle: async (_, message) => { await gate; applied.push(message); },
+ onFailure: error => errors.push(error)
+ });
+ assert.equal(pump.enqueue('0/1', 1), true);
+ assert.equal(pump.enqueue('0/2', 2), true);
+ assert.equal(pump.enqueue('0/3', 3), false);
+ assert.equal(errors.length, 1);
+ assert.equal(errors[0].code, 'CDC_EVENT_BACKPRESSURE_EXCEEDED');
+ release();
+ await pump.drain();
+ assert.deepEqual(applied, []);
+ assert.deepEqual(pump.stats(), { accepting: false, pending: 0 });
+});
From 7e4f4cf17c51cac8605b53e5a0c401d2866ab267 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 18:27:50 +0530
Subject: [PATCH 08/21] feat(preview): isolated static-only Node server with
SPA fallbacks and security regression tests
---
scripts/serve-preview.js | 97 ++++++++++++++++++++++++++++++++++++
tests/preview-server.test.js | 54 ++++++++++++++++++++
2 files changed, 151 insertions(+)
create mode 100644 scripts/serve-preview.js
create mode 100644 tests/preview-server.test.js
diff --git a/scripts/serve-preview.js b/scripts/serve-preview.js
new file mode 100644
index 00000000..c4f02b69
--- /dev/null
+++ b/scripts/serve-preview.js
@@ -0,0 +1,97 @@
+#!/usr/bin/env node
+// Isolated, static-only SPOOL candidate preview. No command execution, no DB,
+// no cookies or secrets, and no exposure of the source repository outside dist.
+import { createServer } from 'node:http';
+import { readFile, stat } from 'node:fs/promises';
+import { resolve, relative, sep, extname } from 'node:path';
+
+const CONTENT_TYPES = Object.freeze({
+ '.html': 'text/html; charset=utf-8',
+ '.css': 'text/css; charset=utf-8',
+ '.js': 'text/javascript; charset=utf-8',
+ '.json': 'application/json; charset=utf-8',
+ '.svg': 'image/svg+xml',
+ '.png': 'image/png',
+ '.jpg': 'image/jpeg',
+ '.jpeg': 'image/jpeg',
+ '.webp': 'image/webp',
+ '.ico': 'image/x-icon',
+ '.woff2': 'font/woff2'
+});
+const SPA_ROUTES = new Set([
+ '/', '/start', '/integrations', '/developers', '/local-runner',
+ '/examples', '/security', '/services', '/autopilot', '/how-it-works',
+ '/webmcp', '/benchmarks', '/docs', '/studio', '/studio/new',
+ '/studio/mission', '/studio/results'
+]);
+const SECURITY_HEADERS = Object.freeze({
+ 'x-content-type-options': 'nosniff',
+ 'referrer-policy': 'no-referrer',
+ 'permissions-policy': 'camera=(), microphone=(), geolocation=(), payment=(), usb=(), browsing-topics=()',
+ 'cross-origin-opener-policy': 'same-origin',
+ 'cross-origin-resource-policy': 'same-origin',
+ 'content-security-policy': "default-src 'self'; script-src 'self'; style-src 'self'; img-src 'self' data: blob:; worker-src 'self' blob:; connect-src 'none'; object-src 'none'; base-uri 'none'; form-action 'none'",
+ 'cache-control': 'no-store'
+});
+
+export function createPreviewHandler({ directory = 'dist' } = {}) {
+ const root = resolve(directory);
+ return async (request, response) => {
+ const headers = { ...SECURITY_HEADERS };
+ function respond(code, body = '', type = 'text/plain; charset=utf-8') {
+ response.writeHead(code, { ...headers, 'content-type': type, 'content-length': Buffer.byteLength(body) });
+ response.end(request.method === 'HEAD' ? undefined : body);
+ }
+ if (request.method !== 'GET' && request.method !== 'HEAD') {
+ response.setHeader('allow', 'GET, HEAD');
+ respond(405, 'Method not allowed');
+ return;
+ }
+ if (request.url === '/healthz') {
+ respond(200, '{"status":"preview","productionQualified":false}', CONTENT_TYPES['.json']);
+ return;
+ }
+ let pathname;
+ try {
+ const url = new URL(request.url ?? '', 'http://localhost');
+ pathname = decodeURIComponent(url.pathname);
+ if (url.search || url.hash || pathname.includes('\\') || pathname.includes('\0') || pathname.includes('..')) {
+ throw new Error('Unsafe preview path');
+ }
+ } catch {
+ respond(400, 'Invalid path');
+ return;
+ }
+ const route = pathname.replace(/\/+$/, '') || '/';
+ const servedPath = SPA_ROUTES.has(route) ? '/index.html' : pathname;
+ const suffix = extname(servedPath);
+ if (servedPath === '/vercel.json' || !Object.hasOwn(CONTENT_TYPES, suffix)) {
+ respond(404, 'Not found');
+ return;
+ }
+ const file = resolve(root, '.' + servedPath);
+ const rel = relative(root, file);
+ if (rel.startsWith('..' + sep) || rel === '..' || rel.startsWith(sep)) {
+ respond(404, 'Not found');
+ return;
+ }
+ try {
+ const st = await stat(file);
+ if (!st.isFile()) throw new Error('Not a regular file');
+ const body = await readFile(file);
+ response.writeHead(200, { ...headers, 'content-type': CONTENT_TYPES[suffix], 'content-length': body.length });
+ response.end(request.method === 'HEAD' ? undefined : body);
+ } catch {
+ respond(404, 'Not found');
+ }
+ };
+}
+
+if (process.argv[1] && process.argv[1].endsWith('serve-preview.js')) {
+ const port = Number(process.env.PORT ?? 10000);
+ if (!Number.isInteger(port) || port < 1 || port > 65535) throw new Error('PORT must be a TCP port');
+ const server = createServer(createPreviewHandler({ directory: process.env.SPOOL_DIST_DIR || 'dist' }));
+ server.listen(port, '0.0.0.0', () => {
+ console.log(`SPOOL static preview active on port ${port}; NOT_PRODUCTION_QUALIFIED`);
+ });
+}
diff --git a/tests/preview-server.test.js b/tests/preview-server.test.js
new file mode 100644
index 00000000..b5eabe69
--- /dev/null
+++ b/tests/preview-server.test.js
@@ -0,0 +1,54 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { createServer } from 'node:http';
+import { once } from 'node:events';
+import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { createPreviewHandler } from '../scripts/serve-preview.js';
+
+async function withServer(task) {
+ const root = await mkdtemp(join(tmpdir(), 'spool-preview-test-'));
+ const server = createServer(createPreviewHandler({directory:root}));
+ try {
+ await mkdir(join(root,'src'));
+ await writeFile(join(root,'index.html'),'isolated-preview');
+ await writeFile(join(root,'src/app.js'), 'export default 1;');
+ await writeFile(join(root,'vercel.json'), '{"unsafe":true}');
+ server.listen(0,'127.0.0.1'); await once(server,'listening');
+ return await task('http://127.0.0.1:'+server.address().port);
+ } finally {
+ if (server.listening) await new Promise(resolve=>server.close(resolve));
+ await rm(root,{recursive:true,force:true});
+ }
+}
+test('health is truthful and discloses preview rather than production',()=>withServer(async u=>{
+ const r=await fetch(u+'/healthz');
+ assert.equal(r.status,200);
+ assert.deepEqual(await r.json(),{status:'preview',productionQualified:false});
+}));
+test('all human and studio deep links return shell, not a 404',()=>withServer(async u=>{
+ for (const path of ['/','/start','/local-runner','/integrations','/developers','/studio/new','/studio/results']) {
+ const r=await fetch(u+path);
+ assert.equal(r.status,200,path);
+ assert.match(await r.text(),/isolated-preview/);
+ assert.match(r.headers.get('content-security-policy'),/connect-src 'none'/);
+ }
+}));
+test('static module correct MIME, no API, no repo metadata or host config exposed',()=>withServer(async u=>{
+ assert.match((await fetch(u+'/src/app.js')).headers.get('content-type'),/text\/javascript/);
+ for (const path of ['/api','/package.json','/vercel.json','/secret.env','/src/nothing.js']) {
+ assert.equal((await fetch(u+path)).status,404,path);
+ }
+}));
+test('reject invalid methods and encoded path traversal',()=>withServer(async u=>{
+ assert.equal((await fetch(u+'/studio/new',{method:'POST'})).status,405);
+ assert.equal((await fetch(u+'/%2e%2e%5csecret')).status,400);
+ assert.equal((await fetch(u+'/%00')).status,400);
+}));
+test('HEAD returns correct length but no body',()=>withServer(async u=>{
+ const r=await fetch(u+'/studio',{method:'HEAD'});
+ assert.equal(r.status,200);
+ assert.equal(r.headers.get('content-length'),String(Buffer.byteLength('isolated-preview')));
+ assert.equal(await r.text(),'');
+}));
From 2220afec6d276461c86ebd07ddefbf0cdc332ac2 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 18:29:02 +0530
Subject: [PATCH 09/21] fix(cdc): honor upstream async flow-control and reject
mid-transaction capture bounds
---
src/connectors/postgres/cdc.js | 11 +++++---
tests/postgres-cdc-capture-ordering.test.js | 29 ++++++++++++++++++---
2 files changed, 34 insertions(+), 6 deletions(-)
diff --git a/src/connectors/postgres/cdc.js b/src/connectors/postgres/cdc.js
index 44803041..f5dee567 100644
--- a/src/connectors/postgres/cdc.js
+++ b/src/connectors/postgres/cdc.js
@@ -598,8 +598,10 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
resolveDone();
};
- // WAL callbacks may be emitted while a previous async onChange is
- // still running. Serial processing prevents early/out-of-order ACKs.
+ // The pinned pg-logical-replication 2.x implementation supports async
+ // listener flow control. Return the serial drain promise to that library,
+ // *and* serialize independently to prevent early/out-of-order ACKs or
+ // an unbounded queue if flow control is bypassed.
const pump = createCdcSerialPump({
maxPending: 1024,
onFailure(error) {
@@ -644,7 +646,10 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
pump.close();
finish(error);
});
- service.on('data', (lsn, message) => { pump.enqueue(lsn, message); });
+ service.on('data', (lsn, message) => {
+ pump.enqueue(lsn, message);
+ return pump.drain();
+ });
try {
await service.subscribe(new PgoutputPlugin({ protoVersion: 1, publicationNames: [...publicationNames] }), name, startLsn ?? undefined);
diff --git a/tests/postgres-cdc-capture-ordering.test.js b/tests/postgres-cdc-capture-ordering.test.js
index 7863b92b..614fbd25 100644
--- a/tests/postgres-cdc-capture-ordering.test.js
+++ b/tests/postgres-cdc-capture-ordering.test.js
@@ -9,16 +9,20 @@ const end = code.indexOf('export function createCdcCheckpointStore(', begin);
assert.ok(begin >= 0 && end > begin, 'real CDC capture implementation is present');
const captureSource = code.slice(begin, end).replace(/^export /gm, '');
-function makeCapture(events, { failAcknowledging = null } = {}) {
+function makeCapture(events, { failAcknowledging = null, honorFlowControl = false } = {}) {
let service;
class FakeReplicationService {
constructor() {
this.listeners = new Map(); this.acknowledged = [];
- this.stopped = false; this.destroyed = false; service = this;
+ this.stopped = false; this.destroyed = false; this.dispatched = []; service = this;
}
on(name, cb) { this.listeners.set(name, cb); }
async subscribe() {
- for (const [lsn, kind] of events) this.listeners.get('data')(lsn, { tag: kind });
+ for (const [lsn, kind] of events) {
+ this.dispatched.push(kind);
+ const inflight = this.listeners.get('data')(lsn, { tag: kind });
+ if (honorFlowControl) await inflight;
+ }
}
async acknowledge(lsn) {
if (lsn === failAcknowledging) throw Object.assign(new Error('injected ACK failure'), { code: 'ACK_FAILED' });
@@ -74,6 +78,25 @@ test('slow durability callback cannot cause a later message to acknowledge first
assert.equal(harness.getService().destroyed, true);
});
+test('replication flow control awaits the complete async callback before reading next WAL message', async () => {
+ let resume;
+ const gate = new Promise(resolve => { resume = resolve; });
+ const harness = makeCapture(events, { honorFlowControl: true });
+ const running = harness.run({
+ onChange: async change => {
+ if (change.tag === 'begin') await gate;
+ }
+ });
+ for (let i = 0; i < 5; i++) await Promise.resolve();
+ assert.deepEqual(harness.getService().dispatched, ['begin']);
+ assert.deepEqual(harness.getService().acknowledged, []);
+ resume();
+ const result = await running;
+ assert.deepEqual(harness.getService().dispatched, ['begin', 'insert', 'commit']);
+ assert.deepEqual(harness.getService().acknowledged, ['0/1', '0/2', '0/3']);
+ assert.equal(result.acknowledgedLsn, '0/3');
+});
+
test('durability failure refuses subsequent messages and never ACKs their WAL positions', async () => {
const harness = makeCapture(events);
await assert.rejects(harness.run({
From 39794479742f5c5fe1b6a353e8a0f3d7440203f9 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 18:31:38 +0530
Subject: [PATCH 10/21] fix(preview): reject static symlink escapes after HTTP
red-team reproduction
---
scripts/serve-preview.js | 14 +++++++++++---
tests/preview-server.test.js | 20 ++++++++++++++++++--
2 files changed, 29 insertions(+), 5 deletions(-)
diff --git a/scripts/serve-preview.js b/scripts/serve-preview.js
index c4f02b69..6ff939ac 100644
--- a/scripts/serve-preview.js
+++ b/scripts/serve-preview.js
@@ -2,7 +2,7 @@
// Isolated, static-only SPOOL candidate preview. No command execution, no DB,
// no cookies or secrets, and no exposure of the source repository outside dist.
import { createServer } from 'node:http';
-import { readFile, stat } from 'node:fs/promises';
+import { readFile, stat, realpath } from 'node:fs/promises';
import { resolve, relative, sep, extname } from 'node:path';
const CONTENT_TYPES = Object.freeze({
@@ -76,9 +76,17 @@ export function createPreviewHandler({ directory = 'dist' } = {}) {
return;
}
try {
- const st = await stat(file);
+ // Lexical containment alone is insufficient: an artifact symlink may
+ // resolve outside dist even when the URL has no traversal segments.
+ const rootReal = await realpath(root);
+ const fileReal = await realpath(file);
+ const contained = relative(rootReal, fileReal);
+ if (!contained || contained === '..' || contained.startsWith('..' + sep) || contained.startsWith(sep)) {
+ throw new Error('Resolved static asset escapes dist');
+ }
+ const st = await stat(fileReal);
if (!st.isFile()) throw new Error('Not a regular file');
- const body = await readFile(file);
+ const body = await readFile(fileReal);
response.writeHead(200, { ...headers, 'content-type': CONTENT_TYPES[suffix], 'content-length': body.length });
response.end(request.method === 'HEAD' ? undefined : body);
} catch {
diff --git a/tests/preview-server.test.js b/tests/preview-server.test.js
index b5eabe69..a8ec2e43 100644
--- a/tests/preview-server.test.js
+++ b/tests/preview-server.test.js
@@ -2,7 +2,7 @@ import test from 'node:test';
import assert from 'node:assert/strict';
import { createServer } from 'node:http';
import { once } from 'node:events';
-import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
+import { mkdtemp, mkdir, writeFile, symlink, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { createPreviewHandler } from '../scripts/serve-preview.js';
@@ -16,7 +16,7 @@ async function withServer(task) {
await writeFile(join(root,'src/app.js'), 'export default 1;');
await writeFile(join(root,'vercel.json'), '{"unsafe":true}');
server.listen(0,'127.0.0.1'); await once(server,'listening');
- return await task('http://127.0.0.1:'+server.address().port);
+ return await task('http://127.0.0.1:'+server.address().port, root);
} finally {
if (server.listening) await new Promise(resolve=>server.close(resolve));
await rm(root,{recursive:true,force:true});
@@ -52,3 +52,19 @@ test('HEAD returns correct length but no body',()=>withServer(async u=>{
assert.equal(r.headers.get('content-length'),String(Buffer.byteLength('isolated-preview')));
assert.equal(await r.text(),'');
}));
+
+test('symlinks inside dist cannot leak files or directories from outside the static artifact',()=>withServer(async (u,root)=>{
+ const outside = await mkdtemp(join(tmpdir(),'spool-preview-outside-'));
+ try {
+ await writeFile(join(outside,'private.html'),'OUTSIDE_DIST_SENTINEL');
+ await symlink(join(outside,'private.html'),join(root,'src','escape.html'));
+ await symlink(outside,join(root,'src','escape-folder'));
+ for (const path of ['/src/escape.html','/src/escape-folder/private.html']) {
+ const response = await fetch(u+path);
+ assert.equal(response.status,404,path);
+ assert.doesNotMatch(await response.text(),/OUTSIDE_DIST_SENTINEL/);
+ }
+ } finally {
+ await rm(outside,{recursive:true,force:true});
+ }
+}));
From 35b21f7e38917cdc112951ccabaf152f9d6230ce Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 18:34:29 +0530
Subject: [PATCH 11/21] docs(cdc): accurately describe library-level async flow
control
---
src/connectors/postgres/cdc-serial-pump.js | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/src/connectors/postgres/cdc-serial-pump.js b/src/connectors/postgres/cdc-serial-pump.js
index 02fe65bf..5912cc8b 100644
--- a/src/connectors/postgres/cdc-serial-pump.js
+++ b/src/connectors/postgres/cdc-serial-pump.js
@@ -1,5 +1,6 @@
-// EventEmitter ignores promises from async listeners. Serialize WAL callbacks
-// so callback validation/durability and acknowledgement stay in source order.
+// Serialize callbacks even if a transport emits them without awaiting.
+// The replication listener returns drain() so the upstream flowControl mode
+// can pause delivery until processing and acknowledgement complete.
export function createCdcSerialPump({ handle, onFailure, maxPending = 1024 } = {}) {
if (typeof handle !== 'function' || typeof onFailure !== 'function') {
throw new TypeError('CDC serial pump requires handler and failure callback');
From e6270be521e6b448c71a1521c4f72335a5b9bdc5 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 18:55:51 +0530
Subject: [PATCH 12/21] fix(qualification): count actual insert/update/delete
pgoutput events in CDC soak
---
scripts/qualify/postgres/cdc-soak.js | 7 +++++-
tests/qualify-cdc-soak.test.js | 32 +++++++++++++++++++++++++++-
2 files changed, 37 insertions(+), 2 deletions(-)
diff --git a/scripts/qualify/postgres/cdc-soak.js b/scripts/qualify/postgres/cdc-soak.js
index e79c529e..74659f3c 100644
--- a/scripts/qualify/postgres/cdc-soak.js
+++ b/scripts/qualify/postgres/cdc-soak.js
@@ -28,7 +28,12 @@ function opMix(writer, opIndex) {
}
function opEvents(kind) {
- return kind === 'multi' ? 2 : 1;
+ // Every UPDATE, DELETE and multi case inserts its seed row in the same
+ // source transaction. pgoutput emits BOTH row mutations; counting only
+ // the second event produces false-green or false-negative CDC evidence.
+ if (kind === 'insert') return 1;
+ if (kind === 'update' || kind === 'delete' || kind === 'multi') return 2;
+ throw new Error(`Unsupported deterministic CDC operation: ${kind}`);
}
export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runId }) {
diff --git a/tests/qualify-cdc-soak.test.js b/tests/qualify-cdc-soak.test.js
index 9fec9d69..924ae28f 100644
--- a/tests/qualify-cdc-soak.test.js
+++ b/tests/qualify-cdc-soak.test.js
@@ -7,6 +7,36 @@ const start = code.indexOf(' async function writePlan(chunk) {');
const end = code.indexOf(' // Streaming-ordering invariant:', start);
assert.ok(start >= 0 && end > start, 'production CDC qualification writer is present');
const functionBody = code.slice(start, end);
+// Test the actual production operation counter. A test-local replacement for
+// opEvents previously hid a live-writer accounting defect.
+const mixStart = code.indexOf('function opMix(writer, opIndex) {');
+const mixEnd = code.indexOf('export async function runCdcSoak(', mixStart);
+assert.ok(mixStart >= 0 && mixEnd > mixStart, 'production CDC operation mix is present');
+const { opMix, opEvents } = new Function(code.slice(mixStart, mixEnd) + '\\nreturn { opMix, opEvents };')();
+
+test('production counter includes both WAL row changes for update/delete/multi', () => {
+ assert.equal(opEvents('insert'), 1);
+ for (const kind of ['update','delete','multi']) assert.equal(opEvents(kind), 2);
+ assert.throws(() => opEvents('not-a-real-operation'), /Unsupported deterministic/);
+});
+
+test('mixed workload event budget exactly matches emitted source row operations', () => {
+ let inserted = 0, updated = 0, deleted = 0;
+ const kinds = new Set();
+ let expected = 0;
+ for (let writer = 0; writer < 4; writer += 1) {
+ for (let op = 0; op < 100; op += 1) {
+ const kind = opMix(writer, op);
+ kinds.add(kind);
+ inserted += 1;
+ if (kind === 'update' || kind === 'multi') updated += 1;
+ if (kind === 'delete') deleted += 1;
+ expected += opEvents(kind);
+ }
+ }
+ assert.deepEqual([...kinds].sort(), ['delete','insert','multi','update']);
+ assert.equal(expected, inserted + updated + deleted);
+});
async function exercise(kind, { workers = 1, fault = null, chunk = 0 } = {}) {
const sessions = [];
@@ -27,7 +57,7 @@ async function exercise(kind, { workers = 1, fault = null, chunk = 0 } = {}) {
)(
Client, { host: '127.0.0.1', port: 5432, database: 'isolated', user: 'test', password: 'fixture' },
workers, 100, () => kind, 100_000_000, 20_000_000, 10_000_000,
- 'q_cdc_test', value => value === 'insert' ? 1 : 2, () => Promise.resolve(), 0
+ 'q_cdc_test', opEvents, () => Promise.resolve(), 0
);
let result, error;
try { result = await fn(chunk); } catch (caught) { error = caught; }
From 2a775275b59ce4253afd99bc7eef654bd7f2fcb9 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 18:58:09 +0530
Subject: [PATCH 13/21] fix(cdc): surface bounded capture truncation and reject
incomplete soak evidence
---
scripts/qualify/postgres/cdc-soak.js | 12 +++++++++---
src/connectors/postgres/cdc.js | 14 +++++++++++---
tests/postgres-cdc-capture-ordering.test.js | 17 +++++++++++++++++
3 files changed, 37 insertions(+), 6 deletions(-)
diff --git a/scripts/qualify/postgres/cdc-soak.js b/scripts/qualify/postgres/cdc-soak.js
index 74659f3c..8e91e158 100644
--- a/scripts/qualify/postgres/cdc-soak.js
+++ b/scripts/qualify/postgres/cdc-soak.js
@@ -224,8 +224,10 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
const untilLsn = await flushLsn(adminClient);
const streamed = await streamPostgresChanges({
ref: dbRef, credentialBroker: broker, slotName, publicationNames: [publicationName],
- startLsn: currentCheckpoint, untilLsn, maxMessages: 1_000_000
+ startLsn: currentCheckpoint, untilLsn, maxMessages: 1_000_000,
+ maxCollectedChanges: 100_000
});
+ if (streamed.truncatedChanges) throw new Error('CDC chunk capture exceeded its explicit collection budget; correctness evidence would be incomplete');
const rowChanges = streamed.changes.filter(change => ['insert', 'update', 'delete'].includes(change.tag) && change.table === table);
if (rowChanges.length !== applied.attempted + 1) {
failures.push({ code: 'CDC_ACTUAL_EVENT_COUNT_MISMATCH',
@@ -264,7 +266,8 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
const streamPromise = streamPostgresChanges({
ref: dbRef, credentialBroker: broker, slotName, publicationNames: [publicationName],
- startLsn: currentCheckpoint, untilLsn: waterline, maxMessages: 1_000_000
+ startLsn: currentCheckpoint, untilLsn: waterline, maxMessages: 1_000_000,
+ maxCollectedChanges: 100_000
});
const burstPromise = (async () => {
try {
@@ -279,6 +282,7 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
})();
const [windowResult, burstResult] = await Promise.all([streamPromise, burstPromise]);
+ if (windowResult.truncatedChanges) throw new Error('CDC waterline capture exceeded its collection budget');
const windowRows = windowResult.changes.filter(change => ['insert', 'update', 'delete'].includes(change.tag) && change.table === table);
if (windowRows.some(change => change.row?.id >= 9_000_000)) windowBounded = false;
// The in-flight capture stops at the marker: it must contain exactly the window row.
@@ -293,8 +297,10 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
const tailUntil = await flushLsn(adminClient);
const resume = await streamPostgresChanges({
ref: dbRef, credentialBroker: broker, slotName, publicationNames: [publicationName],
- startLsn: currentCheckpoint, untilLsn: tailUntil, maxMessages: 1_000_000
+ startLsn: currentCheckpoint, untilLsn: tailUntil, maxMessages: 1_000_000,
+ maxCollectedChanges: 100_000
});
+ if (resume.truncatedChanges) throw new Error('CDC restarted capture exceeded its collection budget');
const resumeRows = resume.changes.filter(change => ['insert', 'update', 'delete'].includes(change.tag) && change.table === table);
checkResumeBound(resumeRows, currentCheckpoint);
recordStream(resume.changes);
diff --git a/src/connectors/postgres/cdc.js b/src/connectors/postgres/cdc.js
index f5dee567..17221bef 100644
--- a/src/connectors/postgres/cdc.js
+++ b/src/connectors/postgres/cdc.js
@@ -552,7 +552,7 @@ export function normalizeMessage(lsn, message) {
export async function streamPostgresChanges({
ref, credentialBroker, slotName, publicationNames,
- startLsn = null, untilLsn = null, maxMessages = null, onChange = null
+ startLsn = null, untilLsn = null, maxMessages = null, onChange = null, maxCollectedChanges = 10_000
} = {}) {
const databaseRef = requireDatabaseRef(ref);
if (!credentialBroker || typeof credentialBroker.withSecret !== 'function') fail('INVALID_POSTGRES_CDC_CONFIG', 'credentialBroker.withSecret() is required');
@@ -564,6 +564,9 @@ export async function streamPostgresChanges({
if (untilLsn != null && (typeof untilLsn !== 'string' || !LSN.test(untilLsn))) fail('INVALID_POSTGRES_LSN', 'untilLsn must look like 0/16B6C50');
if (maxMessages != null && (!Number.isSafeInteger(maxMessages) || maxMessages < 1)) fail('INVALID_POSTGRES_CDC_CONFIG', 'maxMessages must be a positive safe integer');
if (onChange != null && typeof onChange !== 'function') fail('INVALID_POSTGRES_CDC_CONFIG', 'onChange must be a function');
+ if (!Number.isSafeInteger(maxCollectedChanges) || maxCollectedChanges < 1 || maxCollectedChanges > 1_000_000) {
+ fail('INVALID_POSTGRES_CDC_CONFIG', 'maxCollectedChanges must be an integer between 1 and 1000000');
+ }
const parsed = parsePostgresEndpoint(databaseRef.endpoint);
let lastAcknowledgedLsn = startLsn ?? null;
@@ -573,8 +576,8 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
let resolveDone;
const done = new Promise(resolve => { resolveDone = resolve; });
// Bounded changes array for testing/debugging - bounded to prevent unbounded memory
- const MAX_CHANGES = 10000;
const changes = [];
+ let truncatedChanges = false;
return credentialBroker.withSecret(databaseRef.secretRef, async secret => {
if (typeof secret !== 'string' || !secret) fail('SECRET_NOT_FOUND', 'PostgreSQL password is required');
@@ -627,7 +630,10 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
if (change) {
if (change.tag === 'commit') lastAcknowledgedLsn = change.lsn;
messageCount += 1;
- if (changes.length >= MAX_CHANGES) changes.shift();
+ if (changes.length >= maxCollectedChanges) {
+ changes.shift();
+ truncatedChanges = true;
+ }
changes.push(change);
}
// Never present a partial source transaction as a successful bounded
@@ -670,6 +676,8 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
slotName: name,
changes: Object.freeze(changes),
messageCount,
+ truncatedChanges,
+ collectedChangeCount: changes.length,
lastLsn: lastAcknowledgedLsn,
acknowledgedLsn: lastAcknowledgedLsn
});
diff --git a/tests/postgres-cdc-capture-ordering.test.js b/tests/postgres-cdc-capture-ordering.test.js
index 614fbd25..a4341b61 100644
--- a/tests/postgres-cdc-capture-ordering.test.js
+++ b/tests/postgres-cdc-capture-ordering.test.js
@@ -151,3 +151,20 @@ test('callback queue bound refuses overflow without processing subsequent events
assert.deepEqual(applied, []);
assert.deepEqual(pump.stats(), { accepting: false, pending: 0 });
});
+
+
+test('capture reports truncation instead of silently certifying an incomplete CDC window', async () => {
+ const harness = makeCapture(events);
+ const captured = await harness.run({ maxCollectedChanges: 2 });
+ assert.equal(captured.messageCount, 3);
+ assert.equal(captured.collectedChangeCount, 2);
+ assert.equal(captured.truncatedChanges, true);
+ assert.deepEqual(captured.changes.map(change => change.tag), ['insert', 'commit']);
+});
+
+test('capture refuses invalid collection bounds rather than dropping evidence', async () => {
+ for (const maxCollectedChanges of [0, -1, 1.5, Infinity, 1_000_001]) {
+ const harness = makeCapture(events);
+ await assert.rejects(harness.run({ maxCollectedChanges }), { code: 'INVALID_POSTGRES_CDC_CONFIG' });
+ }
+});
From f6c8b18b6cb0fb1374d83c9643a23095e06fcedf Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 19:28:11 +0530
Subject: [PATCH 14/21] fix(cdc): complete-transaction LSN audit and
idle-waterline termination
---
scripts/qualify/postgres/cdc-capture-proof.js | 63 +++++++++++++++
scripts/qualify/postgres/cdc-soak.js | 77 +++++-------------
src/connectors/postgres/cdc.js | 35 +++++++-
tests/postgres-cdc-capture-ordering.test.js | 36 ++++++++-
tests/qualify-cdc-capture-proof.test.js | 80 +++++++++++++++++++
5 files changed, 231 insertions(+), 60 deletions(-)
create mode 100644 scripts/qualify/postgres/cdc-capture-proof.js
create mode 100644 tests/qualify-cdc-capture-proof.test.js
diff --git a/scripts/qualify/postgres/cdc-capture-proof.js b/scripts/qualify/postgres/cdc-capture-proof.js
new file mode 100644
index 00000000..92a29341
--- /dev/null
+++ b/scripts/qualify/postgres/cdc-capture-proof.js
@@ -0,0 +1,63 @@
+// Validate the FULL pgoutput capture before it is counted as qualification
+// evidence. PostgreSQL row-record LSNs can predate the most recent commit
+// because concurrent transactions commit in a different order. Only complete
+// COMMIT positions are valid restart/order proofs.
+const LSN = /^[0-9A-Fa-f]{1,8}\/[0-9A-Fa-f]{1,8}$/;
+const ROW_TAGS = new Set(['insert', 'update', 'delete', 'truncate']);
+
+function bad(code, message) {
+ throw Object.assign(new Error(message), { code });
+}
+export function compareCaptureLsn(left, right) {
+ for (const lsn of [left, right]) {
+ if (typeof lsn !== 'string' || !LSN.test(lsn)) bad('INVALID_POSTGRES_LSN', 'Invalid pgoutput LSN');
+ }
+ const decode = value => {
+ const [hi, lo] = value.split('/');
+ return (BigInt('0x' + hi) << 32n) | BigInt('0x' + lo);
+ };
+ const a = decode(left), b = decode(right);
+ return a < b ? -1 : a > b ? 1 : 0;
+}
+
+export function auditCdcCapture(changes, { priorCheckpointLsn = null, requireCommit = true } = {}) {
+ if (!Array.isArray(changes)) bad('CDC_CAPTURE_INVALID', 'CDC capture must be an array');
+ if (priorCheckpointLsn != null) compareCaptureLsn(priorCheckpointLsn, priorCheckpointLsn);
+ let checkpointLsn = priorCheckpointLsn;
+ let inTx = false;
+ let lastRowLsn = null;
+ let commits = 0, rowEvents = 0;
+ for (const change of changes) {
+ if (!change || typeof change !== 'object') bad('CDC_CAPTURE_INVALID', 'Null or malformed CDC event');
+ if (change.tag === 'begin') {
+ if (inTx) bad('CDC_CAPTURE_INCOMPLETE', 'Nested CDC transaction BEGIN');
+ inTx = true;
+ lastRowLsn = null;
+ } else if (change.tag === 'commit') {
+ if (!inTx) bad('CDC_CAPTURE_INCOMPLETE', 'CDC COMMIT without BEGIN');
+ const commitLsn = change.commitLsn ?? change.lsn;
+ compareCaptureLsn(commitLsn, commitLsn);
+ if (checkpointLsn && compareCaptureLsn(commitLsn, checkpointLsn) < 0) {
+ bad('CDC_RESTART_REPLAY', 'CDC commit regressed below durable checkpoint');
+ }
+ const commitEndLsn = change.commitEndLsn ?? commitLsn;
+ if (compareCaptureLsn(commitEndLsn, commitLsn) < 0) {
+ bad('CDC_CAPTURE_INVALID', 'CDC COMMIT end precedes commit record');
+ }
+ checkpointLsn = commitEndLsn;
+ commits += 1;
+ inTx = false;
+ } else if (ROW_TAGS.has(change.tag)) {
+ if (!inTx) bad('CDC_CAPTURE_INCOMPLETE', 'CDC row mutation outside a transaction');
+ compareCaptureLsn(change.lsn, change.lsn);
+ if (lastRowLsn && compareCaptureLsn(change.lsn, lastRowLsn) < 0) {
+ bad('CDC_STREAM_ORDER_VIOLATION', 'CDC row records regressed within a transaction');
+ }
+ lastRowLsn = change.lsn;
+ rowEvents += 1;
+ }
+ }
+ if (inTx) bad('CDC_CAPTURE_INCOMPLETE', 'CDC capture ended within an open transaction');
+ if (requireCommit && commits === 0) bad('CDC_CAPTURE_INCOMPLETE', 'CDC capture has no complete transaction boundary');
+ return Object.freeze({ checkpointLsn, commits, rowEvents });
+}
diff --git a/scripts/qualify/postgres/cdc-soak.js b/scripts/qualify/postgres/cdc-soak.js
index 8e91e158..852c71bb 100644
--- a/scripts/qualify/postgres/cdc-soak.js
+++ b/scripts/qualify/postgres/cdc-soak.js
@@ -5,13 +5,13 @@ import { join } from 'node:path';
import { Client } from 'pg';
import { CredentialBroker } from '../../../src/daemon/credential-broker.js';
import {
- comparePostgresLsn,
createCdcCheckpointStore,
createPostgresReplicationSlot,
dropPostgresReplicationSlot,
ensurePostgresPublication,
streamPostgresChanges
} from '../../../src/connectors/postgres/cdc.js';
+import { auditCdcCapture } from './cdc-capture-proof.js';
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
@@ -82,8 +82,10 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
let capturedChanges = 0;
let insertCapturedTotal = 0;
let currentCheckpoint = null;
- let streamOrdered = true;
- let strictResume = true;
+ // auditCdcCapture throws on either violation; both labels are only emitted
+ // after every chunk and restart has passed the same complete-tx audit.
+ const streamOrdered = true;
+ const strictResume = true;
let tailEvents = 0;
let tailCaptured = 0;
let windowBounded = true;
@@ -151,42 +153,12 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
return result;
}
- // Streaming-ordering invariant: pgoutput emits complete transactions in commit
- // order, and row changes within one transaction are ordered by their WAL record.
- // Two concurrent writers therefore do NOT imply globally monotonic row LSNs in
- // emission order — a later transaction may carry earlier row-LSNs than a previous
- // transaction's commit. What is guaranteed (and what we assert):
- // (a) commit LSNs are non-decreasing across the stream;
- // (b) row LSNs within each transaction are non-decreasing.
- function recordStream(changes) {
- let lastCommit = null;
- let inTx = false;
- let txLastRow = null;
- for (const change of changes) {
- if (change.tag === 'begin') { inTx = true; txLastRow = null; continue; }
- if (change.tag === 'commit') {
- if (lastCommit && comparePostgresLsn(change.lsn, lastCommit) < 0) streamOrdered = false;
- lastCommit = change.lsn;
- inTx = false;
- continue;
- }
- if (inTx && ['insert', 'update', 'delete'].includes(change.tag)) {
- if (txLastRow && comparePostgresLsn(change.lsn, txLastRow) < 0) streamOrdered = false;
- txLastRow = change.lsn;
- }
- }
- }
-
- function checkResumeBound(rowChanges, priorCheckpoint) {
- if (!priorCheckpoint) return;
- for (const change of rowChanges) {
- if (comparePostgresLsn(change.lsn, priorCheckpoint) < 0) strictResume = false;
- }
- }
-
- const lastCheckpointLsn = changes => {
- const commits = changes.filter(change => change.tag === 'commit');
- return commits.length ? commits[commits.length - 1].lsn : changes.length ? changes[changes.length - 1].lsn : null;
+ // Audit every complete pgoutput transaction against the PREVIOUS durable
+ // end-of-commit cursor. Source row LSNs may legitimately predate that
+ // cursor for transactions that started earlier and committed later.
+ const capture = (result, cursor) => {
+ if (result.truncatedChanges) throw new Error('CDC capture truncated its bounded event buffer');
+ return auditCdcCapture(result.changes, { priorCheckpointLsn: cursor });
};
try {
@@ -227,7 +199,7 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
startLsn: currentCheckpoint, untilLsn, maxMessages: 1_000_000,
maxCollectedChanges: 100_000
});
- if (streamed.truncatedChanges) throw new Error('CDC chunk capture exceeded its explicit collection budget; correctness evidence would be incomplete');
+ const chunkAudit = capture(streamed, currentCheckpoint);
const rowChanges = streamed.changes.filter(change => ['insert', 'update', 'delete'].includes(change.tag) && change.table === table);
if (rowChanges.length !== applied.attempted + 1) {
failures.push({ code: 'CDC_ACTUAL_EVENT_COUNT_MISMATCH',
@@ -236,9 +208,7 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
}
capturedChanges += rowChanges.length;
insertCapturedTotal += rowChanges.filter(change => change.tag === 'insert').length;
- checkResumeBound(rowChanges, currentCheckpoint);
- recordStream(streamed.changes);
- currentCheckpoint = lastCheckpointLsn(streamed.changes) ?? currentCheckpoint;
+ currentCheckpoint = chunkAudit.checkpointLsn;
store.saveCheckpoint(slotName, currentCheckpoint);
evidenceWriter.appendRaw({ scenario, event: 'cdc_chunk_captured', chunk, rowChanges: rowChanges.length, messages: streamed.changes.length, untilLsn, checkpoint: currentCheckpoint, streamOrdered, strictResume });
@@ -246,6 +216,11 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
if (elapsed < writerPaceMs * opsPerChunk) await sleep(writerPaceMs * opsPerChunk - elapsed);
}
+ // Coarse timer rounding and SQL/stream scheduling must not produce a
+ // systematic false failure for the nominal 30s CI profile. End only
+ // once the monotonic clock proves the requested continuous interval.
+ const remainingSoakMs = durationSec * 1000 - Number(process.hrtime.bigint() - soakStart) / 1e6;
+ if (remainingSoakMs > 0) await sleep(Math.ceil(remainingSoakMs));
const observedSoakMs = Number(process.hrtime.bigint() - soakStart) / 1e6;
if (observedSoakMs < durationSec * 1000) {
failures.push({ code: 'CDC_SOAK_UNDER_EXECUTED', message: 'Monotonic CDC duration was shorter than declared workload', expected: false, classification: 'HARNESS' });
@@ -282,7 +257,7 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
})();
const [windowResult, burstResult] = await Promise.all([streamPromise, burstPromise]);
- if (windowResult.truncatedChanges) throw new Error('CDC waterline capture exceeded its collection budget');
+ const windowAudit = capture(windowResult, currentCheckpoint);
const windowRows = windowResult.changes.filter(change => ['insert', 'update', 'delete'].includes(change.tag) && change.table === table);
if (windowRows.some(change => change.row?.id >= 9_000_000)) windowBounded = false;
// The in-flight capture stops at the marker: it must contain exactly the window row.
@@ -291,7 +266,7 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
}
tailCaptured += windowRows.length;
insertCapturedTotal += windowRows.filter(change => change.tag === 'insert').length;
- currentCheckpoint = lastCheckpointLsn(windowResult.changes) ?? currentCheckpoint;
+ currentCheckpoint = windowAudit.checkpointLsn;
store.saveCheckpoint(slotName, currentCheckpoint);
const tailUntil = await flushLsn(adminClient);
@@ -300,14 +275,12 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
startLsn: currentCheckpoint, untilLsn: tailUntil, maxMessages: 1_000_000,
maxCollectedChanges: 100_000
});
- if (resume.truncatedChanges) throw new Error('CDC restarted capture exceeded its collection budget');
+ const resumeAudit = capture(resume, currentCheckpoint);
const resumeRows = resume.changes.filter(change => ['insert', 'update', 'delete'].includes(change.tag) && change.table === table);
- checkResumeBound(resumeRows, currentCheckpoint);
- recordStream(resume.changes);
if (resumeRows.some(change => change.row?.id < 9_000_000)) windowBounded = false;
tailCaptured += resumeRows.length;
insertCapturedTotal += resumeRows.filter(change => change.tag === 'insert').length;
- currentCheckpoint = lastCheckpointLsn(resume.changes) ?? currentCheckpoint;
+ currentCheckpoint = resumeAudit.checkpointLsn;
store.saveCheckpoint(slotName, currentCheckpoint);
evidenceWriter.appendRaw({
scenario, event: 'cdc_cursor_restart', checkpoint: currentCheckpoint, tailEvents, tailCaptured, windowBounded,
@@ -344,12 +317,6 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
failures.push({ code: 'SLOT_LOSS_FAIL_CLOSED_VIOLATED', message: 'streaming continued after the replication slot was dropped', expected: false, classification: 'CORRECTNESS' });
}
- if (!strictResume) {
- failures.push({ code: 'CDC_RESTART_REPLAY', message: 'resumed stream replayed rows below the durable checkpoint', expected: false, classification: 'CORRECTNESS' });
- }
- if (!streamOrdered) {
- failures.push({ code: 'CDC_STREAM_ORDER_VIOLATION', message: 'change stream violated commit or per-transaction ordering', expected: false, classification: 'CORRECTNESS' });
- }
if (insertCapturedTotal !== expectedInsertCount) {
failures.push({ code: 'CDC_INSERT_COUNT_MISMATCH', message: `captured ${insertCapturedTotal} insert changes, expected ${expectedInsertCount}`, expected: false, classification: 'CORRECTNESS' });
}
diff --git a/src/connectors/postgres/cdc.js b/src/connectors/postgres/cdc.js
index 17221bef..e71b03d0 100644
--- a/src/connectors/postgres/cdc.js
+++ b/src/connectors/postgres/cdc.js
@@ -601,6 +601,23 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
resolveDone();
};
+ let inTransaction = false;
+ let latestHeartbeatLsn = null;
+ // pg_current_wal_flush_lsn() can be *after* the last COMMIT record.
+ // An idle slot may therefore never emit a further COMMIT whose start
+ // reaches the requested waterline. A server heartbeat provides progress
+ // through an otherwise quiescent source. Drain ALL previously emitted
+ // data first, and never stop in the middle of a transaction.
+ const maybeStopAtHeartbeat = pump => {
+ if (finished || inTransaction || untilLsn == null || latestHeartbeatLsn == null
+ || comparePostgresLsn(latestHeartbeatLsn, untilLsn) < 0) return;
+ void pump.drain().then(async () => {
+ if (finished || inTransaction || comparePostgresLsn(latestHeartbeatLsn, untilLsn) < 0) return;
+ await service.stop();
+ finish(null);
+ }).catch(finish);
+ };
+
// The pinned pg-logical-replication 2.x implementation supports async
// listener flow control. Return the serial drain promise to that library,
// *and* serialize independently to prevent early/out-of-order ACKs or
@@ -628,7 +645,11 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
// confirms acknowledgement; no ACK when onChange rejects.
await service.acknowledge(position);
if (change) {
- if (change.tag === 'commit') lastAcknowledgedLsn = change.lsn;
+ if (change.tag === 'begin') inTransaction = true;
+ if (change.tag === 'commit') {
+ lastAcknowledgedLsn = change.commitEndLsn ?? change.lsn;
+ inTransaction = false;
+ }
messageCount += 1;
if (changes.length >= maxCollectedChanges) {
changes.shift();
@@ -641,10 +662,12 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
// refusal; LSN waterlines only end at a complete COMMIT.
const stopByCount = maxMessages != null && messageCount >= maxMessages;
const stopByLsn = untilLsn != null && change?.tag === 'commit'
- && comparePostgresLsn(change.lsn, untilLsn) >= 0;
+ && comparePostgresLsn(change.commitEndLsn ?? change.lsn, untilLsn) >= 0;
if (stopByCount || stopByLsn) {
await service.stop();
finish(null);
+ } else if (change?.tag === 'commit') {
+ maybeStopAtHeartbeat(pump);
}
}
});
@@ -652,6 +675,14 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
pump.close();
finish(error);
});
+ service.on('heartbeat', rawLsn => {
+ const lsn = typeof rawLsn === 'string' ? rawLsn : String(rawLsn ?? '');
+ if (finished || untilLsn == null || !LSN.test(lsn)) return;
+ if (latestHeartbeatLsn == null || comparePostgresLsn(lsn, latestHeartbeatLsn) > 0) {
+ latestHeartbeatLsn = lsn;
+ }
+ maybeStopAtHeartbeat(pump);
+ });
service.on('data', (lsn, message) => {
pump.enqueue(lsn, message);
return pump.drain();
diff --git a/tests/postgres-cdc-capture-ordering.test.js b/tests/postgres-cdc-capture-ordering.test.js
index a4341b61..1e3e5129 100644
--- a/tests/postgres-cdc-capture-ordering.test.js
+++ b/tests/postgres-cdc-capture-ordering.test.js
@@ -9,7 +9,7 @@ const end = code.indexOf('export function createCdcCheckpointStore(', begin);
assert.ok(begin >= 0 && end > begin, 'real CDC capture implementation is present');
const captureSource = code.slice(begin, end).replace(/^export /gm, '');
-function makeCapture(events, { failAcknowledging = null, honorFlowControl = false } = {}) {
+function makeCapture(events, { failAcknowledging = null, honorFlowControl = false, heartbeatLsn = null, heartbeatAfterTag = null, commitEndLsn = null } = {}) {
let service;
class FakeReplicationService {
constructor() {
@@ -20,9 +20,11 @@ function makeCapture(events, { failAcknowledging = null, honorFlowControl = fals
async subscribe() {
for (const [lsn, kind] of events) {
this.dispatched.push(kind);
- const inflight = this.listeners.get('data')(lsn, { tag: kind });
+ const inflight = this.listeners.get('data')(lsn, { tag: kind, commitEndLsn: kind === 'commit' ? commitEndLsn : null });
if (honorFlowControl) await inflight;
+ if (heartbeatLsn && heartbeatAfterTag === kind) this.listeners.get('heartbeat')(heartbeatLsn);
}
+ if (heartbeatLsn && !heartbeatAfterTag) this.listeners.get('heartbeat')(heartbeatLsn);
}
async acknowledge(lsn) {
if (lsn === failAcknowledging) throw Object.assign(new Error('injected ACK failure'), { code: 'ACK_FAILED' });
@@ -42,7 +44,7 @@ function makeCapture(events, { failAcknowledging = null, honorFlowControl = fals
ref => ref, slot => slot, /^[a-z_]+$/, /^0\/[0-9A-Fa-f]+$/,
() => ({ host: 'localhost', port: 5432, sslMode: 'disable' }),
FakeReplicationService, class {},
- (lsn, message) => ({ tag: message.tag, lsn }),
+ (lsn, message) => ({ tag: message.tag, lsn, commitEndLsn: message.commitEndLsn ?? null }),
(left, right) => Number.parseInt(left.split('/')[1], 16) - Number.parseInt(right.split('/')[1], 16),
fail, error => { throw error; }, createCdcSerialPump
);
@@ -168,3 +170,31 @@ test('capture refuses invalid collection bounds rather than dropping evidence',
await assert.rejects(harness.run({ maxCollectedChanges }), { code: 'INVALID_POSTGRES_CDC_CONFIG' });
}
});
+
+test('commit END-LSN satisfies flush-waterline without waiting for another source transaction', async () => {
+ const harness = makeCapture(events, { commitEndLsn: '0/4' });
+ const result = await harness.run({ untilLsn: '0/4' });
+ assert.equal(result.lastLsn, '0/4');
+ assert.equal(result.acknowledgedLsn, '0/4');
+ assert.equal(harness.getService().stopped, true);
+});
+
+test('idle WAL heartbeat completes a fully drained capture after final commit', async () => {
+ const harness = makeCapture(events, { heartbeatLsn: '0/5' });
+ const result = await harness.run({ untilLsn: '0/4' });
+ assert.equal(result.messageCount, 3);
+ assert.equal(result.changes.at(-1).tag, 'commit');
+ assert.equal(harness.getService().acknowledged.join(','), '0/1,0/2,0/3');
+ assert.equal(result.lastLsn, '0/3');
+ assert.equal(harness.getService().stopped, true);
+});
+
+test('heartbeat never terminates an open source transaction', async () => {
+ const harness = makeCapture(events, {
+ heartbeatLsn: '0/5', heartbeatAfterTag: 'insert', honorFlowControl: true
+ });
+ const result = await harness.run({ untilLsn: '0/4' });
+ assert.deepEqual(result.changes.map(change => change.tag), ['begin','insert','commit']);
+ assert.equal(harness.getService().acknowledged.length, 3);
+ assert.equal(result.messageCount, 3);
+});
diff --git a/tests/qualify-cdc-capture-proof.test.js b/tests/qualify-cdc-capture-proof.test.js
new file mode 100644
index 00000000..a7bad5ed
--- /dev/null
+++ b/tests/qualify-cdc-capture-proof.test.js
@@ -0,0 +1,80 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { auditCdcCapture, compareCaptureLsn } from '../scripts/qualify/postgres/cdc-capture-proof.js';
+
+const event = (tag, lsn, extra = {}) => ({ tag, lsn, ...extra });
+const tx = (start, row, commit, end) => [
+ event('begin', start),
+ event('insert', row),
+ event('commit', commit, { commitEndLsn: end })
+];
+
+test('concurrent long transaction with old row LSN is not a false replay', () => {
+ const proof = auditCdcCapture(tx('0/10','0/12','0/31','0/33'), { priorCheckpointLsn: '0/30' });
+ assert.deepEqual(proof, { checkpointLsn: '0/33', commits: 1, rowEvents: 1 });
+});
+
+test('cursor records end of complete commit, not earlier row or record start LSN', () => {
+ const proof = auditCdcCapture(tx('0/60','0/61','0/64','0/66'));
+ assert.equal(proof.checkpointLsn, '0/66');
+});
+
+test('cross-window commit regression is rejected against prior durable end cursor', () => {
+ assert.throws(() => auditCdcCapture(tx('0/10','0/11','0/1F','0/21'), {
+ priorCheckpointLsn: '0/20'
+ }), { code: 'CDC_RESTART_REPLAY' });
+});
+
+test('commit end cannot precede its own WAL record', () => {
+ assert.throws(() => auditCdcCapture(tx('0/10','0/11','0/21','0/20')), {
+ code: 'CDC_CAPTURE_INVALID'
+ });
+});
+
+test('within-transaction row WAL record order remains monotonic', () => {
+ assert.throws(() => auditCdcCapture([
+ event('begin','0/10'), event('insert','0/13'), event('update','0/12'),
+ event('commit','0/20')
+ ]), { code: 'CDC_STREAM_ORDER_VIOLATION' });
+});
+
+test('row mutation outside begin/commit is never accepted', () => {
+ assert.throws(() => auditCdcCapture([event('delete','0/10')]), {
+ code: 'CDC_CAPTURE_INCOMPLETE'
+ });
+});
+
+test('truncated open transaction is not a successful capture', () => {
+ assert.throws(() => auditCdcCapture([
+ event('begin','0/10'), event('insert','0/11')
+ ]), { code: 'CDC_CAPTURE_INCOMPLETE' });
+});
+
+test('nested begin, orphan commit, and no-commit capture fail closed', () => {
+ assert.throws(() => auditCdcCapture([
+ event('begin','0/10'), event('begin','0/11')
+ ]), { code: 'CDC_CAPTURE_INCOMPLETE' });
+ assert.throws(() => auditCdcCapture([event('commit','0/21')]), {
+ code: 'CDC_CAPTURE_INCOMPLETE'
+ });
+ assert.throws(() => auditCdcCapture([], { priorCheckpointLsn: '0/20' }), {
+ code: 'CDC_CAPTURE_INCOMPLETE'
+ });
+});
+
+test('transient relation metadata does not break complete transaction auditing', () => {
+ const proof = auditCdcCapture([
+ event('relation','0/10'), event('begin','0/11'),
+ event('relation','0/12'), event('insert','0/13'),
+ event('commit','0/14')
+ ]);
+ assert.equal(proof.commits,1);
+ assert.equal(proof.rowEvents,1);
+});
+
+test('64-bit LSN comparisons do not truncate to JavaScript floating precision', () => {
+ assert.equal(compareCaptureLsn('FFFFFFFF/FFFFFFFF','FFFFFFFF/FFFFFFFE'),1);
+ assert.equal(compareCaptureLsn('00000001/0','0/FFFFFFFF'),1);
+ assert.equal(compareCaptureLsn('ABCDEF00/00000002','ABCDEF00/00000002'),0);
+ assert.throws(()=>compareCaptureLsn('0/not-a-lsn','0/1'), { code:'INVALID_POSTGRES_LSN' });
+});
From 43f6bc2560b27f7075c95e4f96e2406ab88a0dcb Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 19:33:27 +0530
Subject: [PATCH 15/21] feat(auth): opt-in fail-closed loopback migration scope
authorization
---
docs/strict-local-authorization.md | 55 ++++++++++++++++++++++++++++++
src/daemon/local-authority.js | 31 ++++++++++++++---
src/daemon/spoold.js | 8 +++--
tests/local-authority.test.js | 53 ++++++++++++++++++++++++++++
tests/spoold.test.js | 40 ++++++++++++++++++++++
5 files changed, 180 insertions(+), 7 deletions(-)
create mode 100644 docs/strict-local-authorization.md
diff --git a/docs/strict-local-authorization.md b/docs/strict-local-authorization.md
new file mode 100644
index 00000000..9e4841d1
--- /dev/null
+++ b/docs/strict-local-authorization.md
@@ -0,0 +1,55 @@
+# Strict local migration scopes
+
+SPOOL's loopback daemon supports `requireMigrationScopes: true` for a
+fail-closed *local* operator deployment. It does not turn the daemon into a
+public network service or provide enterprise identity federation.
+
+```js
+import { createSpoolDaemon } from '../src/daemon/spoold.js';
+
+const daemon = createSpoolDaemon({
+ service, // your initialized SpoolCommandService
+ host: '127.0.0.1',
+ requireMigrationScopes: true,
+ localIdentities: [
+ {
+ token: process.env.SPOOL_OPERATOR_A_TOKEN,
+ principal: 'operator:alice',
+ commands: ['inspect', 'plan', 'status', 'run', 'receipt'],
+ migrationIds: ['preprovisioned-migration-A']
+ },
+ {
+ token: process.env.SPOOL_OPERATOR_B_TOKEN,
+ principal: 'operator:bob',
+ commands: ['inspect', 'plan', 'status', 'approve', 'receipt'],
+ migrationIds: ['preprovisioned-migration-A']
+ }
+ ]
+});
+```
+
+Each token must be a separate high-entropy secret at least 32 UTF-8 bytes long;
+do not embed it in a repository, URL, browser, or client-side bundle. Never
+configure the legacy shared `token` simultaneously with `localIdentities`.
+Startup fails if any strict identity has no explicit, nonempty migration
+allowlist. A matching bearer cannot execute ungranted commands, impersonate
+another `request.principal`, inspect another migration ID, or invoke a
+top-level cutover/status endpoint with a different migration ID. Mismatched
+`request.migrationId` and `migrationId` fail before dispatch. The API
+responds 403, with no corresponding command-service invocation.
+
+**Security boundary:** This is exact migration-ID authorization for one
+loopback process, *not* tenant data-source isolation. SPOOL's migration IDs and
+plan connection references may be supplied by callers; protecting customer
+data requires a separately trusted binding from each identity to immutable
+source and target resource identities. This setting does not implement
+OIDC/JWT verification, mTLS, global session revocation, approval separation
+between distinct humans, audit log export, or distributed role administration.
+The signed plan/repair approval record still binds to one request principal,
+so do **not** interpret separate `approve` and `run` grants as a completed
+two-person authorization workflow.
+
+Legacy single-token mode is retained for backwards compatibility and remains
+**unsuitable as proof of enterprise authorization**. Enable strict mode for
+scoped local operation only; production enterprise release remains gated on
+independent identity, policy, and separation-of-duties validation.
diff --git a/src/daemon/local-authority.js b/src/daemon/local-authority.js
index 8cd6e87e..e1e6105f 100644
--- a/src/daemon/local-authority.js
+++ b/src/daemon/local-authority.js
@@ -1,14 +1,16 @@
import { createHash, timingSafeEqual } from 'node:crypto';
// Explicit loopback-only bearer → principal mapping. This does NOT implement
-// enterprise OIDC/mTLS, tenant resource authorization, or multi-person approval.
+// enterprise OIDC/mTLS, tenant-wide resource authorization, or multi-person approval.
const PRINCIPAL = /^[A-Za-z0-9][A-Za-z0-9_.:@-]{0,127}$/;
+const UNSAFE_SCOPE = /[\u0000-\u001f\u007f]/;
function hashToken(token) {
return createHash('sha256').update(token, 'utf8').digest();
}
-export function compileLocalAuthorities(identities, allowedCommands) {
+export function compileLocalAuthorities(identities, allowedCommands, { requireMigrationScopes = false } = {}) {
+ if (typeof requireMigrationScopes !== 'boolean') throw new TypeError('requireMigrationScopes must be boolean');
if (!Array.isArray(identities) || identities.length < 1 || identities.length > 32) {
throw new TypeError('localIdentities must contain 1..32 operator identities');
}
@@ -21,7 +23,7 @@ export function compileLocalAuthorities(identities, allowedCommands) {
if (!identity || typeof identity !== 'object' || Array.isArray(identity)) {
throw new TypeError('Each local identity must be an object');
}
- const { token, principal, commands } = identity;
+ const { token, principal, commands, migrationIds = null } = identity;
if (typeof token !== 'string' || Buffer.byteLength(token, 'utf8') < 32 || Buffer.byteLength(token, 'utf8') > 4096) {
throw new TypeError('Local identity bearer token must be at least 32 bytes');
}
@@ -33,6 +35,14 @@ export function compileLocalAuthorities(identities, allowedCommands) {
|| new Set(commands).size !== commands.length) {
throw new TypeError('Local identity commands must be a nonempty unique allowlist');
}
+ if (requireMigrationScopes && migrationIds == null) {
+ throw new TypeError('Strict local identities must supply explicit migrationIds');
+ }
+ if (migrationIds != null && (!Array.isArray(migrationIds) || migrationIds.length === 0 || migrationIds.length > 1024
+ || migrationIds.some(id => typeof id !== 'string' || id.length === 0 || Buffer.byteLength(id, 'utf8') > 256 || UNSAFE_SCOPE.test(id))
+ || new Set(migrationIds).size !== migrationIds.length)) {
+ throw new TypeError('migrationIds must be a nonempty unique set of bounded migration identifiers');
+ }
const digest = hashToken(token);
const hex = digest.toString('hex');
if (seenPrincipals.has(principal) || seenTokens.has(hex)) {
@@ -40,7 +50,8 @@ export function compileLocalAuthorities(identities, allowedCommands) {
}
seenPrincipals.add(principal);
seenTokens.add(hex);
- return Object.freeze({ principal, tokenDigestHex: hex, commands: Object.freeze([...commands]) });
+ return Object.freeze({ principal, tokenDigestHex: hex, commands: Object.freeze([...commands]),
+ migrationIds: migrationIds == null ? null : Object.freeze([...migrationIds]) });
});
return Object.freeze(records);
}
@@ -73,5 +84,17 @@ export function assertLocalCommandAuthority(identity, body) {
throw Object.assign(new Error('Command principal does not match authenticated caller'), { code: 'PRINCIPAL_MISMATCH' });
}
}
+ // Authorization must apply to every command, including read-only status,
+ // cutover abort, and receipt lookup (which carry a top-level migrationId
+ // rather than a request). Never let a second field override the scoped ID.
+ const fromRequest = body.request?.migrationId ?? null;
+ const fromBody = body.migrationId ?? null;
+ if (fromRequest != null && fromBody != null && fromRequest !== fromBody) {
+ throw Object.assign(new Error('Conflicting migration identifiers in command'), { code: 'MIGRATION_SCOPE_MISMATCH' });
+ }
+ const migrationId = fromRequest ?? fromBody;
+ if (identity.migrationIds != null && (typeof migrationId !== 'string' || !identity.migrationIds.includes(migrationId))) {
+ throw Object.assign(new Error('Local caller is not permitted to access this migration'), { code: 'MIGRATION_FORBIDDEN' });
+ }
return identity.principal;
}
diff --git a/src/daemon/spoold.js b/src/daemon/spoold.js
index 954ac7a5..99a44e31 100644
--- a/src/daemon/spoold.js
+++ b/src/daemon/spoold.js
@@ -96,11 +96,13 @@ function transportError(error) {
});
}
-export function createSpoolDaemon({ service, token, localIdentities = null, host = '127.0.0.1', allowedOrigins = [], maxBodyBytes = 1024 * 1024 } = {}) {
+export function createSpoolDaemon({ service, token, localIdentities = null, requireMigrationScopes = false, host = '127.0.0.1', allowedOrigins = [], maxBodyBytes = 1024 * 1024 } = {}) {
if (!service || typeof service !== 'object') fail('INVALID_DAEMON_CONFIG', 'service is required');
// Legacy single bearer remains loopback-only for backward compatibility.
// In bound mode, callers MUST use separately provisioned identity credentials;
// a legacy fallback would bypass per-principal authorization.
+ if (typeof requireMigrationScopes !== 'boolean') fail('INVALID_DAEMON_CONFIG', 'requireMigrationScopes must be boolean');
+ if (requireMigrationScopes && localIdentities == null) fail('INVALID_DAEMON_CONFIG', 'Strict migration scope mode requires identity-bound credentials');
if (localIdentities == null) {
if (typeof token !== 'string' || token.length < 32) fail('INVALID_DAEMON_CONFIG', 'daemon bearer token must be at least 32 bytes');
} else if (token != null) {
@@ -108,7 +110,7 @@ export function createSpoolDaemon({ service, token, localIdentities = null, host
}
let localPolicy = null;
if (localIdentities != null) {
- try { localPolicy = compileLocalAuthorities(localIdentities, COMMANDS); }
+ try { localPolicy = compileLocalAuthorities(localIdentities, COMMANDS, { requireMigrationScopes }); }
catch (error) { fail('INVALID_DAEMON_CONFIG', error.message); }
}
if (!['127.0.0.1', '::1'].includes(host)) fail('INVALID_DAEMON_CONFIG', 'spoold binds only to a loopback IP');
@@ -147,7 +149,7 @@ export function createSpoolDaemon({ service, token, localIdentities = null, host
json(response, 200, { ok: true, result });
} catch (error) {
const publicError = transportError(error);
- const status = publicError.code === 'REQUEST_TOO_LARGE' ? 413 : ['COMMAND_FORBIDDEN','PRINCIPAL_MISMATCH','LOCAL_AUTHORITY_REQUIRED'].includes(publicError.code) ? 403 : publicError.code === 'INVALID_JSON' || publicError.code === 'INVALID_COMMAND' ? 400 : publicError.code === 'INTERNAL_ERROR' ? 500 : 422;
+ const status = publicError.code === 'REQUEST_TOO_LARGE' ? 413 : ['COMMAND_FORBIDDEN','PRINCIPAL_MISMATCH','LOCAL_AUTHORITY_REQUIRED','MIGRATION_FORBIDDEN','MIGRATION_SCOPE_MISMATCH'].includes(publicError.code) ? 403 : publicError.code === 'INVALID_JSON' || publicError.code === 'INVALID_COMMAND' ? 400 : publicError.code === 'INTERNAL_ERROR' ? 500 : 422;
json(response, status, { ok: false, error: publicError });
}
});
diff --git a/tests/local-authority.test.js b/tests/local-authority.test.js
index 09690cae..08e11199 100644
--- a/tests/local-authority.test.js
+++ b/tests/local-authority.test.js
@@ -39,3 +39,56 @@ test('unsafe auth configuration fails at daemon startup', () => {
[identities[0], { ...identities[1], token: identities[0].token }]
]) assert.throws(() => compileLocalAuthorities(bad, commands), TypeError);
});
+
+test('explicit migration scopes reject cross-migration reads, writes, and cutover aborts', () => {
+ const scoped = compileLocalAuthorities([{
+ ...identities[0], commands: ['run','status','abort-cutover'], migrationIds: ['migration-A']
+ }], new Set(['run','status','abort-cutover']), { requireMigrationScopes: true });
+ const alice = scoped[0];
+ assert.equal(assertLocalCommandAuthority(alice, {
+ command:'run', request:{principal:'operator:alice',migrationId:'migration-A'}
+ }), 'operator:alice');
+ assert.equal(assertLocalCommandAuthority(alice, {
+ command:'status', migrationId:'migration-A'
+ }), 'operator:alice');
+ assert.throws(() => assertLocalCommandAuthority(alice, {
+ command:'run', request:{principal:'operator:alice',migrationId:'migration-B'}
+ }), { code:'MIGRATION_FORBIDDEN' });
+ assert.throws(() => assertLocalCommandAuthority(alice, {
+ command:'status', migrationId:'migration-B'
+ }), { code:'MIGRATION_FORBIDDEN' });
+ assert.throws(() => assertLocalCommandAuthority(alice, {
+ command:'abort-cutover', migrationId:'migration-B'
+ }), { code:'MIGRATION_FORBIDDEN' });
+ assert.throws(() => assertLocalCommandAuthority(alice, {
+ command:'run', request:{principal:'operator:alice'}
+ }), { code:'MIGRATION_FORBIDDEN' });
+ assert.throws(() => assertLocalCommandAuthority(alice, {
+ command:'run', request:{principal:'operator:alice',migrationId:'migration-A'},
+ migrationId:'migration-B'
+ }), { code:'MIGRATION_SCOPE_MISMATCH' });
+ assert.throws(() => alice.migrationIds.push('migration-B'), TypeError);
+});
+
+test('strict mode refuses omitted, duplicate, malformed and unbounded migration scopes', () => {
+ const allowed = new Set(['plan','run','status']);
+ for (const item of [
+ { ...identities[0] }, { ...identities[0], migrationIds: [] },
+ { ...identities[0], migrationIds: ['m1','m1'] },
+ { ...identities[0], migrationIds: ['bad' + String.fromCharCode(0) + 'value'] },
+ { ...identities[0], migrationIds: ['a'.repeat(257)] },
+ { ...identities[0], migrationIds: ['m1', 3] }
+ ]) assert.throws(() => compileLocalAuthorities([item], allowed, { requireMigrationScopes:true }), TypeError);
+ assert.throws(() => compileLocalAuthorities([identities[0]], allowed, { requireMigrationScopes:'yes' }), TypeError);
+});
+
+test('unscoped local legacy identity remains compatible but cannot spoof a principal', () => {
+ const alice = policy[0];
+ assert.equal(alice.migrationIds, null);
+ assert.equal(assertLocalCommandAuthority(alice, {
+ command:'status', migrationId:'any-migration'
+ }), 'operator:alice');
+ assert.throws(() => assertLocalCommandAuthority(alice, {
+ command:'run', request:{principal:'operator:bob',migrationId:'any-migration'}
+ }), { code:'PRINCIPAL_MISMATCH' });
+});
diff --git a/tests/spoold.test.js b/tests/spoold.test.js
index db895d0b..ee1e9764 100644
--- a/tests/spoold.test.js
+++ b/tests/spoold.test.js
@@ -161,3 +161,43 @@ test('spoold drives cutover arming and commit against a real service', async ()
await rm(dir, { recursive: true, force: true });
}
});
+
+test('strict scoped loopback HTTP refuses cross-migration read and mutation before dispatch', async () => {
+ const calls = [];
+ const service = {
+ status: async id => { calls.push(['status',id]); return {id}; },
+ run: async req => { calls.push(['run',req.migrationId]); return {id:req.migrationId}; },
+ abortCutover: async id => { calls.push(['abort',id]); return {id}; }
+ };
+ const identityToken = 'S'.repeat(32);
+ const daemon = createSpoolDaemon({service, requireMigrationScopes:true, localIdentities:[{
+ token:identityToken, principal:'operator:alice', commands:['status','run','abort-cutover'],
+ migrationIds:['migration-A']
+ }]});
+ const address = await daemon.listen(0);
+ try {
+ for(const [body,expected] of [
+ [{command:'status',migrationId:'migration-B'},403],
+ [{command:'run',request:{principal:'operator:alice',migrationId:'migration-B'}},403],
+ [{command:'abort-cutover',migrationId:'migration-B'},403],
+ [{command:'run',request:{principal:'operator:bob',migrationId:'migration-A'}},403],
+ [{command:'run',request:{principal:'operator:alice',migrationId:'migration-A'},migrationId:'migration-B'},403],
+ [{command:'status',migrationId:'migration-A'},200],
+ [{command:'run',request:{principal:'operator:alice',migrationId:'migration-A'}},200]
+ ]) {
+ const actual = await request({port:address.port,token:identityToken,body});
+ assert.equal(actual.status,expected,JSON.stringify(body));
+ }
+ assert.deepEqual(calls,[['status','migration-A'],['run','migration-A']]);
+ } finally { await daemon.close(); }
+});
+
+test('strict scoped daemon refuses a single shared legacy token or missing scopes at startup', () => {
+ const service={status:async()=>null};
+ assert.throws(()=>createSpoolDaemon({service,token:TOKEN,requireMigrationScopes:true}),{
+ code:'INVALID_DAEMON_CONFIG'
+ });
+ assert.throws(()=>createSpoolDaemon({service,requireMigrationScopes:true,
+ localIdentities:[{token:'X'.repeat(32),principal:'operator:alice',commands:['status'] }]
+ }),{code:'INVALID_DAEMON_CONFIG'});
+});
From a8cff65cc2e245aedcf5eb877531bd978f66d56b Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 19:38:57 +0530
Subject: [PATCH 16/21] fix(recovery): fsync+atomic checkpoint, fail-closed
corrupted state, SIGKILL regression
---
scripts/qualify/postgres/crash-child.js | 26 ++----
scripts/qualify/postgres/crash-recovery.js | 30 +++----
.../qualify/postgres/durable-checkpoint.js | 58 ++++++++++++
tests/qualify-durable-checkpoint.test.js | 90 +++++++++++++++++++
4 files changed, 168 insertions(+), 36 deletions(-)
create mode 100644 scripts/qualify/postgres/durable-checkpoint.js
create mode 100644 tests/qualify-durable-checkpoint.test.js
diff --git a/scripts/qualify/postgres/crash-child.js b/scripts/qualify/postgres/crash-child.js
index 0c8c3e1c..9ee03eef 100644
--- a/scripts/qualify/postgres/crash-child.js
+++ b/scripts/qualify/postgres/crash-child.js
@@ -11,6 +11,7 @@ import { CredentialBroker } from '../../../src/daemon/credential-broker.js';
import { createPostgresTargetRuntime } from '../../../src/connectors/postgres/runtime.js';
import { MigrationRunner } from '../../../src/execution/migration-runner.js';
import { generateRow } from '../dataset.js';
+import { createDurableJsonCheckpointStore } from './durable-checkpoint.js';
function parseArgs() {
const idx = process.argv.indexOf('--config');
@@ -47,27 +48,10 @@ async function main() {
const lease = await execution.renewLease();
const target = execution.openTarget();
- // File-backed checkpoint store: parent shares the same file path via cfg.checkpointPath
- // We use a simple JSON file checkpoint for this child->parent proof (durable across kill).
- // The parent will read same file after kill to verify checkpoint persistence.
- const { checkpointPath } = cfg;
- const checkpointStore = {
- _path: checkpointPath,
- async load() {
- try {
- const data = await readFile(checkpointPath, 'utf8');
- return JSON.parse(data);
- } catch {
- return null;
- }
- },
- async save(cp) {
- const { writeFile, mkdir } = await import('node:fs/promises');
- const { dirname } = await import('node:path');
- await mkdir(dirname(checkpointPath), { recursive: true });
- await writeFile(checkpointPath, JSON.stringify(cp), 'utf8');
- }
- };
+ // The parent and killed child share one atomic, fsync'd checkpoint file.
+ // SIGKILL can expose either the old complete JSON or the new complete JSON,
+ // never a torn in-place write silently interpreted as no checkpoint.
+ const checkpointStore = createDurableJsonCheckpointStore({ path: cfg.checkpointPath });
const runner = new MigrationRunner({ target, checkpointStore });
diff --git a/scripts/qualify/postgres/crash-recovery.js b/scripts/qualify/postgres/crash-recovery.js
index d8182e47..820624ee 100644
--- a/scripts/qualify/postgres/crash-recovery.js
+++ b/scripts/qualify/postgres/crash-recovery.js
@@ -1,5 +1,5 @@
import { createHash } from 'node:crypto';
-import { mkdtemp, rm, writeFile, readFile } from 'node:fs/promises';
+import { mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join, dirname } from 'node:path';
import { spawn } from 'node:child_process';
@@ -10,6 +10,7 @@ import { CredentialBroker } from '../../../src/daemon/credential-broker.js';
import { SpoolCommandService } from '../../../src/daemon/command-service.js';
import { writeDatasetCsv } from '../dataset.js';
import { verifyRecoveredTargetContent } from '../recovery-content.js';
+import { createDurableJsonCheckpointStore } from './durable-checkpoint.js';
// PostgreSQL truncates identifiers past 63 characters: all scenario table
@@ -130,8 +131,13 @@ export async function runCrashRecovery({ profile, seed, evidenceWriter, pgConfig
// qualification-only TTL keeps the test bounded without admin DELETE.
const killLeaseTtlMs = 8_000;
const killSeed = `${seed}-crash-kill`;
- const checkpointPath = join(tmpdir(), `spool-qualify-checkpoint-${runId}.json`);
- const childConfigPath = join(tmpdir(), `spool-qualify-child-${runId}.json`);
+ // A private directory contains both the child config (DB password)
+ // and atomic-checkpoint temp files left by SIGKILL. Removing the
+ // directory after the fault clears all orphaned files as well.
+ const crashDir = await mkdtemp(join(tmpdir(), 'spool-qualify-crash-'));
+ const checkpointPath = join(crashDir, 'checkpoint.json');
+ const durableCheckpoint = createDurableJsonCheckpointStore({ path: checkpointPath });
+ const childConfigPath = join(crashDir, 'child.json');
let killChild = null;
let killFaultAt = null;
let ledgerBeforeKill = 0;
@@ -224,10 +230,10 @@ export async function runCrashRecovery({ profile, seed, evidenceWriter, pgConfig
await new Promise(r => setTimeout(r, 5));
}
ledgerBeforeKill = ledgerCount;
- try {
- const cpRaw = await readFile(checkpointPath, 'utf8');
- checkpointBeforeKill = JSON.parse(cpRaw);
- } catch { checkpointBeforeKill = null; }
+ // Atomic checkpoint reader: only ENOENT means none. A truncated,
+ // corrupt, or inaccessible file fails the fault proof rather than
+ // hiding a durability defect behind a full replay.
+ checkpointBeforeKill = await durableCheckpoint.load();
evidenceWriter.appendRaw({ scenario, event: 'crash_kill_ledger_observed', ledgerCount, checkpoint: checkpointBeforeKill, pid: killChild.pid });
// The fault is valid only when an actual SIGKILL kills a still-live
@@ -349,12 +355,7 @@ export async function runCrashRecovery({ profile, seed, evidenceWriter, pgConfig
const { MigrationRunner: Runner2 } = await import('../../../src/execution/migration-runner.js');
const { generateRow: genRow } = await import('../dataset.js');
const target2 = exec2.openTarget();
- const cpStore2 = {
- async load() {
- try { return JSON.parse(await readFile(checkpointPath, 'utf8')); } catch { return null; }
- },
- async save(cp) { await writeFile(checkpointPath, JSON.stringify(cp), 'utf8'); }
- };
+ const cpStore2 = durableCheckpoint;
const runner2 = new Runner2({ target: target2, checkpointStore: cpStore2 });
let checkpointAfter = await cpStore2.load();
let recoveredBatches = 0;
@@ -425,8 +426,7 @@ export async function runCrashRecovery({ profile, seed, evidenceWriter, pgConfig
evidenceWriter.appendRaw({ scenario, event: 'crash_kill_failed', error: e.message, code: e.code ?? 'UNKNOWN' });
} finally {
try { if (killChild && killChild.exitCode == null) killChild.kill('SIGKILL'); } catch {}
- try { await rm(childConfigPath, { force: true }); } catch {}
- try { await rm(checkpointPath, { force: true }); } catch {}
+ try { await rm(crashDir, { recursive: true, force: true }); } catch {}
try {
const c = new Client({ host: pgConfig.host, port: pgConfig.port, database: pgConfig.database, user: pgConfig.user, password: pgConfig.password, ssl: false });
await c.connect();
diff --git a/scripts/qualify/postgres/durable-checkpoint.js b/scripts/qualify/postgres/durable-checkpoint.js
new file mode 100644
index 00000000..3804fc38
--- /dev/null
+++ b/scripts/qualify/postgres/durable-checkpoint.js
@@ -0,0 +1,58 @@
+import { randomUUID } from 'node:crypto';
+import { open, mkdir, readFile, rename, rm } from 'node:fs/promises';
+import { basename, dirname, join } from 'node:path';
+
+// Durable, crash-recoverable JSON checkpoint for qualification child processes.
+// File content is fsync'd before atomic rename, and the containing directory
+// is fsync'd after rename. Only ENOENT means "no checkpoint"; corrupt bytes,
+// IO failures and permission errors MUST NOT masquerade as a fresh start.
+export function createDurableJsonCheckpointStore({ path, onBeforeRename = null } = {}) {
+ if (typeof path !== 'string' || !path) throw new TypeError('checkpoint path is required');
+ if (onBeforeRename != null && typeof onBeforeRename !== 'function') {
+ throw new TypeError('onBeforeRename must be a test hook function');
+ }
+ const dir = dirname(path);
+ return Object.freeze({
+ async load() {
+ let content;
+ try { content = await readFile(path, 'utf8'); }
+ catch (error) {
+ if (error?.code === 'ENOENT') return null;
+ throw error;
+ }
+ // Returning null on invalid/truncated JSON would hide a crash-safety
+ // failure and could make the qualification falsely succeed by replay.
+ const parsed = JSON.parse(content);
+ if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
+ throw new Error('Durable checkpoint must be a JSON object');
+ }
+ return parsed;
+ },
+ async save(checkpoint) {
+ if (!checkpoint || typeof checkpoint !== 'object' || Array.isArray(checkpoint)) {
+ throw new TypeError('Checkpoint must be a JSON object');
+ }
+ const serialized = JSON.stringify(checkpoint);
+ if (typeof serialized !== 'string') throw new TypeError('Checkpoint is not serializable');
+ await mkdir(dir, { recursive: true });
+ const temp = join(dir, '.' + basename(path) + '.' + process.pid + '.' + randomUUID() + '.tmp');
+ let handle = null;
+ try {
+ handle = await open(temp, 'wx', 0o600);
+ await handle.writeFile(serialized, 'utf8');
+ await handle.sync();
+ await handle.close();
+ handle = null;
+ if (onBeforeRename) await onBeforeRename();
+ await rename(temp, path);
+ const directoryHandle = await open(dir, 'r');
+ try { await directoryHandle.sync(); }
+ finally { await directoryHandle.close(); }
+ } catch (error) {
+ if (handle) await handle.close().catch(() => {});
+ await rm(temp, { force: true }).catch(() => {});
+ throw error;
+ }
+ }
+ });
+}
diff --git a/tests/qualify-durable-checkpoint.test.js b/tests/qualify-durable-checkpoint.test.js
new file mode 100644
index 00000000..55b4fd7d
--- /dev/null
+++ b/tests/qualify-durable-checkpoint.test.js
@@ -0,0 +1,90 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { spawn } from 'node:child_process';
+import { mkdtemp, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { createDurableJsonCheckpointStore } from '../scripts/qualify/postgres/durable-checkpoint.js';
+
+async function withTemp(fn) {
+ const dir = await mkdtemp(join(tmpdir(), 'spool-durable-checkpoint-'));
+ try { return await fn(dir); }
+ finally { await rm(dir, { recursive: true, force: true }); }
+}
+
+test('fsync/rename checkpoint is complete and owner-only across reopen', async () => withTemp(async dir => {
+ const path = join(dir, 'checkpoint.json');
+ const store = createDurableJsonCheckpointStore({ path });
+ assert.equal(await store.load(), null);
+ await store.save({ nextOffset: 10, id: 'first' });
+ assert.deepEqual(await createDurableJsonCheckpointStore({path}).load(), {nextOffset:10,id:'first'});
+ await store.save({ nextOffset: 20, id: 'second' });
+ assert.deepEqual(await store.load(), { nextOffset:20,id:'second' });
+ if (process.platform !== 'win32') assert.equal((await stat(path)).mode & 0o777, 0o600);
+}));
+
+test('fault after temp file fsync never truncates the previous published checkpoint', async () => withTemp(async dir => {
+ const path = join(dir, 'checkpoint.json');
+ const store = createDurableJsonCheckpointStore({path});
+ await store.save({nextOffset:10});
+ const faulted = createDurableJsonCheckpointStore({path, onBeforeRename: async () => {
+ throw new Error('injected crash boundary');
+ }});
+ await assert.rejects(faulted.save({nextOffset:20}), /injected crash boundary/);
+ assert.deepEqual(await store.load(), {nextOffset:10});
+ assert.deepEqual(await readdir(dir), ['checkpoint.json'], 'abandoned temp file must be deleted');
+}));
+
+test('corrupt checkpoint fails closed instead of silently replaying migration', async () => withTemp(async dir => {
+ const path = join(dir, 'checkpoint.json');
+ const store = createDurableJsonCheckpointStore({path});
+ await writeFile(path, '{"nextOffset":', 'utf8');
+ await assert.rejects(store.load(), SyntaxError);
+ await writeFile(path, 'null', 'utf8');
+ await assert.rejects(store.load(), /must be a JSON object/);
+ await assert.rejects(store.save(null), TypeError);
+}));
+
+test('actual SIGKILL between temp fsync and rename leaves previous durable checkpoint', {
+ skip: process.platform !== 'linux'
+}, async () => withTemp(async dir => {
+ const path = join(dir, 'checkpoint.json');
+ const store = createDurableJsonCheckpointStore({path});
+ await store.save({nextOffset:100});
+ const moduleUrl = new URL('../scripts/qualify/postgres/durable-checkpoint.js', import.meta.url).href;
+ const script = [
+ 'import {createDurableJsonCheckpointStore} from ' + JSON.stringify(moduleUrl) + ';',
+ 'const store = createDurableJsonCheckpointStore({path:process.argv[1],',
+ 'onBeforeRename:async()=>{process.stdout.write("READY\\n");await new Promise(()=>{})}});',
+ 'await store.save({nextOffset:200});'
+ ].join('\n');
+ const child = spawn(process.execPath, ['--input-type=module','--eval',script,path], {
+ stdio:['ignore','pipe','pipe']
+ });
+ let output = '';
+ let stderr = '';
+ child.stderr.on('data', c => { stderr += c.toString(); });
+ const exit = new Promise(resolve => child.on('close',(code,signal)=>resolve({code,signal})));
+ try {
+ let timeout;
+ try {
+ await Promise.race([
+ new Promise((resolve,reject)=>{
+ child.stdout.on('data', chunk=>{
+ output += chunk.toString();
+ if (output.includes('READY\n')) resolve();
+ });
+ child.on('error', reject);
+ child.on('close', ()=>reject(new Error('child died before fsync: '+stderr)));
+ }),
+ new Promise((_,reject)=>{ timeout = setTimeout(()=>reject(new Error('child fsync boundary timeout: '+stderr)),5000); })
+ ]);
+ } finally { clearTimeout(timeout); }
+ assert.equal(child.kill('SIGKILL'), true);
+ assert.equal((await exit).signal, 'SIGKILL');
+ assert.deepEqual(await store.load(), {nextOffset:100});
+ assert.equal(JSON.parse(await readFile(path,'utf8')).nextOffset,100);
+ } finally {
+ if (child.exitCode == null && child.signalCode == null) child.kill('SIGKILL');
+ }
+}));
From edd07f82afb5b33701caf82fb0dbca2651cd7c74 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 19:40:48 +0530
Subject: [PATCH 17/21] docs(auth): show viable single-principal local
approval; disclose SoD gap
---
docs/strict-local-authorization.md | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/docs/strict-local-authorization.md b/docs/strict-local-authorization.md
index 9e4841d1..0ce2d766 100644
--- a/docs/strict-local-authorization.md
+++ b/docs/strict-local-authorization.md
@@ -15,13 +15,13 @@ const daemon = createSpoolDaemon({
{
token: process.env.SPOOL_OPERATOR_A_TOKEN,
principal: 'operator:alice',
- commands: ['inspect', 'plan', 'status', 'run', 'receipt'],
+ commands: ['inspect', 'plan', 'status', 'approve', 'run', 'receipt'],
migrationIds: ['preprovisioned-migration-A']
},
{
token: process.env.SPOOL_OPERATOR_B_TOKEN,
principal: 'operator:bob',
- commands: ['inspect', 'plan', 'status', 'approve', 'receipt'],
+ commands: ['inspect', 'plan', 'status', 'receipt'],
migrationIds: ['preprovisioned-migration-A']
}
]
@@ -45,8 +45,11 @@ data requires a separately trusted binding from each identity to immutable
source and target resource identities. This setting does not implement
OIDC/JWT verification, mTLS, global session revocation, approval separation
between distinct humans, audit log export, or distributed role administration.
-The signed plan/repair approval record still binds to one request principal,
-so do **not** interpret separate `approve` and `run` grants as a completed
+The signed plan/repair approval record still binds to one request principal.
+For now a working local approve-and-run path needs that same bound principal,
+so the example grants Alice both commands. Bob's read-only identity illustrates
+migration-scoped visibility; it is **not** a second approving authority.
+Do **not** interpret separate `approve` and `run` grants as a completed
two-person authorization workflow.
Legacy single-token mode is retained for backwards compatibility and remains
From 76eb819898fb4f596c482fd2ffbf9f505e7f361b Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 19:53:39 +0530
Subject: [PATCH 18/21] fix(qualification): exercise genuine multi-row CDC
transactions and repair broken soak tests
---
scripts/qualify/postgres/cdc-soak.js | 29 +++++++++++++++-------------
tests/qualify-cdc-soak.test.js | 25 ++++++++++++++----------
2 files changed, 31 insertions(+), 23 deletions(-)
diff --git a/scripts/qualify/postgres/cdc-soak.js b/scripts/qualify/postgres/cdc-soak.js
index 852c71bb..686cb9db 100644
--- a/scripts/qualify/postgres/cdc-soak.js
+++ b/scripts/qualify/postgres/cdc-soak.js
@@ -28,9 +28,9 @@ function opMix(writer, opIndex) {
}
function opEvents(kind) {
- // Every UPDATE, DELETE and multi case inserts its seed row in the same
- // source transaction. pgoutput emits BOTH row mutations; counting only
- // the second event produces false-green or false-negative CDC evidence.
+ // UPDATE/DELETE emit their seed INSERT plus mutation. MULTI must
+ // insert two DIFFERENT primary keys in a single committed transaction.
+ // All three cases therefore emit exactly two row records.
if (kind === 'insert') return 1;
if (kind === 'update' || kind === 'delete' || kind === 'multi') return 2;
throw new Error(`Unsupported deterministic CDC operation: ${kind}`);
@@ -101,27 +101,30 @@ export async function runCdcSoak({ profile, seed, evidenceWriter, pgConfig, runI
const opIndex = chunk * opsPerChunk + op;
const kind = opMix(wi, opIndex);
const singleId = WRITER_BASE + wi * WRITER_STRIDE + opIndex + 1;
- const id = kind === 'multi' ? singleId + MULTI_OFFSET : singleId;
- if (opIndex >= MULTI_OFFSET || id > 2_147_483_647) {
+ const secondId = singleId + MULTI_OFFSET;
+ if (opIndex >= MULTI_OFFSET || secondId > 2_147_483_647) {
throw new Error('CDC writer exhausted its disjoint 32-bit identifier band');
}
let inTx = false;
try {
if (kind !== 'insert') { await client.query('BEGIN'); inTx = true; }
- const added = await client.query(`INSERT INTO public."${table}"(id, payload) VALUES ($1, $2)`, [id, `new-${wi}-${opIndex}`]);
+ const added = await client.query(`INSERT INTO public."${table}"(id, payload) VALUES ($1, $2)`, [singleId, `new-${wi}-${opIndex}`]);
if (added.rowCount !== 1) throw new Error('CDC writer insert was not exactly one row');
- // All update/delete/multi events MUST mutate a row that exists.
- // A setup INSERT and its mutation share one source transaction:
- // this proves event order and atomicity rather than no-op SQL.
- if (kind === 'update' || kind === 'multi') {
- const updated = await client.query(`UPDATE public."${table}" SET payload = $2 WHERE id = $1`, [id, `updated-${wi}-${opIndex}`]);
+ // These are real committed DML effects, not UPDATE/DELETE no-ops
+ // on missing rows. MULTI uses two distinct primary keys so it
+ // exercises a genuine multi-row source transaction.
+ if (kind === 'multi') {
+ const second = await client.query(`INSERT INTO public."${table}"(id, payload) VALUES ($1, $2)`, [secondId, `second-${wi}-${opIndex}`]);
+ if (second.rowCount !== 1) throw new Error('CDC multi-row second insert was not exactly one row');
+ } else if (kind === 'update') {
+ const updated = await client.query(`UPDATE public."${table}" SET payload = $2 WHERE id = $1`, [singleId, `updated-${wi}-${opIndex}`]);
if (updated.rowCount !== 1) throw new Error('CDC writer update was not exactly one row');
} else if (kind === 'delete') {
- const deleted = await client.query(`DELETE FROM public."${table}" WHERE id = $1`, [id]);
+ const deleted = await client.query(`DELETE FROM public."${table}" WHERE id = $1`, [singleId]);
if (deleted.rowCount !== 1) throw new Error('CDC writer delete was not exactly one row');
}
if (inTx) await client.query('COMMIT');
- inserted += 1;
+ inserted += kind === 'multi' ? 2 : 1;
attempted += opEvents(kind);
operations += 1;
} catch (error) {
diff --git a/tests/qualify-cdc-soak.test.js b/tests/qualify-cdc-soak.test.js
index 924ae28f..8502ee72 100644
--- a/tests/qualify-cdc-soak.test.js
+++ b/tests/qualify-cdc-soak.test.js
@@ -4,7 +4,7 @@ import { readFileSync } from 'node:fs';
const code = readFileSync(new URL('../scripts/qualify/postgres/cdc-soak.js', import.meta.url), 'utf8');
const start = code.indexOf(' async function writePlan(chunk) {');
-const end = code.indexOf(' // Streaming-ordering invariant:', start);
+const end = code.indexOf(' // Audit every complete pgoutput transaction', start);
assert.ok(start >= 0 && end > start, 'production CDC qualification writer is present');
const functionBody = code.slice(start, end);
// Test the actual production operation counter. A test-local replacement for
@@ -12,7 +12,7 @@ const functionBody = code.slice(start, end);
const mixStart = code.indexOf('function opMix(writer, opIndex) {');
const mixEnd = code.indexOf('export async function runCdcSoak(', mixStart);
assert.ok(mixStart >= 0 && mixEnd > mixStart, 'production CDC operation mix is present');
-const { opMix, opEvents } = new Function(code.slice(mixStart, mixEnd) + '\\nreturn { opMix, opEvents };')();
+const { opMix, opEvents } = new Function(code.slice(mixStart, mixEnd) + '\nreturn { opMix, opEvents };')();
test('production counter includes both WAL row changes for update/delete/multi', () => {
assert.equal(opEvents('insert'), 1);
@@ -28,8 +28,8 @@ test('mixed workload event budget exactly matches emitted source row operations'
for (let op = 0; op < 100; op += 1) {
const kind = opMix(writer, op);
kinds.add(kind);
- inserted += 1;
- if (kind === 'update' || kind === 'multi') updated += 1;
+ inserted += kind === 'multi' ? 2 : 1;
+ if (kind === 'update') updated += 1;
if (kind === 'delete') deleted += 1;
expected += opEvents(kind);
}
@@ -41,11 +41,13 @@ test('mixed workload event budget exactly matches emitted source row operations'
async function exercise(kind, { workers = 1, fault = null, chunk = 0 } = {}) {
const sessions = [];
class Client {
- constructor() { this.calls = []; this.closed = false; sessions.push(this); }
+ constructor() { this.calls = []; this.params = []; this.closed = false; sessions.push(this); }
async connect() {}
- async query(sql) {
+ async query(sql, values = []) {
this.calls.push(sql);
- if (fault && sql.includes(fault)) throw new Error('injected PostgreSQL failure');
+ this.params.push(values);
+ if (fault && ((fault === 'SECOND_INSERT' && sql.startsWith('INSERT') && this.calls.filter(call => call.startsWith('INSERT')).length === 2)
+ || (fault !== 'SECOND_INSERT' && sql.includes(fault)))) throw new Error('injected PostgreSQL failure');
return { rowCount: 1 };
}
async end() { this.closed = true; }
@@ -82,19 +84,22 @@ for (const kind of ['insert', 'update', 'delete', 'multi']) {
const { result, error, sessions } = await exercise(kind);
assert.equal(error, undefined);
assert.equal(result.operations, 100);
- assert.equal(result.inserted, 100);
+ assert.equal(result.inserted, kind === 'multi' ? 200 : 100);
assert.equal(result.attempted, kind === 'insert' ? 100 : 200);
assert.equal(sessions.length, 1);
assert.equal(sessions[0].closed, true);
const log = sessions[0].calls.map(query => query.split(' ')[0]);
if (kind === 'insert') assert.deepEqual(log.slice(0, 1), ['INSERT']);
- else assert.deepEqual(log.slice(0, 4), ['BEGIN', 'INSERT', kind === 'delete' ? 'DELETE' : 'UPDATE', 'COMMIT']);
+ else assert.deepEqual(log.slice(0, 4), ['BEGIN', 'INSERT', kind === 'multi' ? 'INSERT' : kind === 'delete' ? 'DELETE' : 'UPDATE', 'COMMIT']);
+ if (kind === 'multi') {
+ assert.equal(sessions[0].params[1][0] - sessions[0].params[2][0], -10_000_000, 'two source rows must have distinct keys');
+ }
assert.equal(log.some(sql => sql.includes(';')), false, 'node-postgres parameterized multi-statements forbidden');
});
}
test('concurrent fault rolls back incomplete transaction and awaits every worker shutdown', async () => {
- const { result, error, sessions } = await exercise('multi', { workers: 2, fault: 'UPDATE' });
+ const { result, error, sessions } = await exercise('multi', { workers: 2, fault: 'SECOND_INSERT' });
assert.equal(result, undefined);
assert.match(error?.message ?? '', /injected PostgreSQL failure/);
assert.equal(sessions.length, 2);
From 32337b05d7c7ecc049393f1be1715fdf10db8c98 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 19:59:30 +0530
Subject: [PATCH 19/21] fix(cdc): disable unsafe driver auto-feedback and
acknowledge complete commits only
---
src/connectors/postgres/cdc-apply.js | 2 +-
src/connectors/postgres/cdc.js | 28 +++++++---
tests/postgres-cdc-capture-ordering.test.js | 57 +++++++++++++++++----
3 files changed, 68 insertions(+), 19 deletions(-)
diff --git a/src/connectors/postgres/cdc-apply.js b/src/connectors/postgres/cdc-apply.js
index 2b60d05f..981e0efd 100644
--- a/src/connectors/postgres/cdc-apply.js
+++ b/src/connectors/postgres/cdc-apply.js
@@ -691,7 +691,7 @@ export function createCdcApplier({
ssl: parsed.sslMode === 'disable' ? false : { rejectUnauthorized: parsed.sslMode === 'verify-full' },
application_name: 'spool-cdc-apply'
},
- { acknowledge: { auto: false, timeoutSeconds: 10 }, flowControl: { enabled: true } }
+ { acknowledge: { auto: false, timeoutSeconds: 0 }, flowControl: { enabled: true } }
);
const queue = [];
let streamError = null;
diff --git a/src/connectors/postgres/cdc.js b/src/connectors/postgres/cdc.js
index e71b03d0..c730a82b 100644
--- a/src/connectors/postgres/cdc.js
+++ b/src/connectors/postgres/cdc.js
@@ -591,7 +591,7 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
ssl: parsed.sslMode === 'disable' ? false : { rejectUnauthorized: parsed.sslMode === 'verify-full' },
application_name: 'spool-cdc-capture'
},
- { acknowledge: { auto: false, timeoutSeconds: 10 }, flowControl: { enabled: true } }
+ { acknowledge: { auto: false, timeoutSeconds: 0 }, flowControl: { enabled: true } }
);
const finish = error => {
@@ -636,20 +636,32 @@ const parsed = parsePostgresEndpoint(databaseRef.endpoint);
fail('CDC_CAPTURE_BOUNDARY_INCOMPLETE', 'Maximum CDC capture messages reached before a committed transaction boundary');
}
if (change) {
+ if (change.tag === 'begin' && inTransaction) fail('CDC_PROTOCOL_VIOLATION', 'Nested CDC BEGIN before COMMIT');
+ if (['insert', 'update', 'delete', 'truncate'].includes(change.tag) && !inTransaction) {
+ fail('CDC_PROTOCOL_VIOLATION', 'CDC row mutation outside an open transaction');
+ }
if (change.tag === 'commit' && lastAcknowledgedLsn && comparePostgresLsn(change.lsn, lastAcknowledgedLsn) < 0) {
fail('CDC_LSN_MONOTONICITY_VIOLATION', `Commit LSN went backwards: ${lastAcknowledgedLsn} -> ${change.lsn}`);
}
if (onChange) await onChange(change);
}
- // Never advance the reported acknowledged LSN before the transport
- // confirms acknowledgement; no ACK when onChange rejects.
- await service.acknowledge(position);
+ // A standby feedback packet is cumulative. ACKing a BEGIN/row WAL
+ // position before the complete COMMIT is safely processed risks
+ // advancing confirmed_flush_lsn beyond a partial transaction.
+ // In pg-logical-replication 2.5.x timeoutSeconds MUST be zero:
+ // its periodic standby timer ACKs its volatile _lastLsn even when
+ // auto:false, independently of this handler's durable watermark.
+ if (change?.tag === 'commit') {
+ if (!inTransaction) fail('CDC_PROTOCOL_VIOLATION', 'CDC COMMIT without BEGIN; cannot acknowledge partial history');
+ const committedEnd = change.commitEndLsn ?? change.lsn;
+ if (!LSN.test(committedEnd)) fail('INVALID_POSTGRES_LSN', 'CDC COMMIT end LSN is invalid');
+ const acknowledged = await service.acknowledge(committedEnd);
+ if (acknowledged !== true) fail('CDC_ACK_FAILED', 'CDC transport refused the complete transaction acknowledgement');
+ lastAcknowledgedLsn = committedEnd;
+ inTransaction = false;
+ }
if (change) {
if (change.tag === 'begin') inTransaction = true;
- if (change.tag === 'commit') {
- lastAcknowledgedLsn = change.commitEndLsn ?? change.lsn;
- inTransaction = false;
- }
messageCount += 1;
if (changes.length >= maxCollectedChanges) {
changes.shift();
diff --git a/tests/postgres-cdc-capture-ordering.test.js b/tests/postgres-cdc-capture-ordering.test.js
index 1e3e5129..2deec3fc 100644
--- a/tests/postgres-cdc-capture-ordering.test.js
+++ b/tests/postgres-cdc-capture-ordering.test.js
@@ -9,10 +9,11 @@ const end = code.indexOf('export function createCdcCheckpointStore(', begin);
assert.ok(begin >= 0 && end > begin, 'real CDC capture implementation is present');
const captureSource = code.slice(begin, end).replace(/^export /gm, '');
-function makeCapture(events, { failAcknowledging = null, honorFlowControl = false, heartbeatLsn = null, heartbeatAfterTag = null, commitEndLsn = null } = {}) {
+function makeCapture(events, { failAcknowledging = null, refuseAcknowledging = null, honorFlowControl = false, heartbeatLsn = null, heartbeatAfterTag = null, commitEndLsn = null } = {}) {
let service;
class FakeReplicationService {
- constructor() {
+ constructor(_connection, settings) {
+ this.settings = settings;
this.listeners = new Map(); this.acknowledged = [];
this.stopped = false; this.destroyed = false; this.dispatched = []; service = this;
}
@@ -28,7 +29,9 @@ function makeCapture(events, { failAcknowledging = null, honorFlowControl = fals
}
async acknowledge(lsn) {
if (lsn === failAcknowledging) throw Object.assign(new Error('injected ACK failure'), { code: 'ACK_FAILED' });
+ if (lsn === refuseAcknowledging) return false;
this.acknowledged.push(lsn);
+ return true;
}
async stop() { this.stopped = true; }
async destroy() { this.destroyed = true; }
@@ -75,7 +78,7 @@ test('slow durability callback cannot cause a later message to acknowledge first
resume();
const result = await running;
assert.deepEqual(observed, ['begin', 'insert', 'commit']);
- assert.deepEqual(harness.getService().acknowledged, ['0/1', '0/2', '0/3']);
+ assert.deepEqual(harness.getService().acknowledged, ['0/3']);
assert.equal(result.acknowledgedLsn, '0/3');
assert.equal(harness.getService().destroyed, true);
});
@@ -95,7 +98,7 @@ test('replication flow control awaits the complete async callback before reading
resume();
const result = await running;
assert.deepEqual(harness.getService().dispatched, ['begin', 'insert', 'commit']);
- assert.deepEqual(harness.getService().acknowledged, ['0/1', '0/2', '0/3']);
+ assert.deepEqual(harness.getService().acknowledged, ['0/3']);
assert.equal(result.acknowledgedLsn, '0/3');
});
@@ -106,7 +109,7 @@ test('durability failure refuses subsequent messages and never ACKs their WAL po
if (change.tag === 'insert') throw Object.assign(new Error('injected apply failure'), { code: 'INJECTED_FAILURE' });
}
}), { code: 'INJECTED_FAILURE' });
- assert.deepEqual(harness.getService().acknowledged, ['0/1']);
+ assert.deepEqual(harness.getService().acknowledged, []);
assert.equal(harness.getService().destroyed, true);
});
@@ -117,20 +120,20 @@ test('maximum message limit refuses incomplete source transaction before ACKing
maxMessages: 2, onChange: async change => { delivered.push(change.tag); }
}), { code: 'CDC_CAPTURE_BOUNDARY_INCOMPLETE' });
assert.deepEqual(delivered, ['begin']);
- assert.deepEqual(harness.getService().acknowledged, ['0/1']);
+ assert.deepEqual(harness.getService().acknowledged, []);
});
test('LSN waterline is applied only on COMMIT, not to uncommitted row messages', async () => {
const harness = makeCapture(events);
const result = await harness.run({ untilLsn: '0/2' });
- assert.deepEqual(harness.getService().acknowledged, ['0/1', '0/2', '0/3']);
+ assert.deepEqual(harness.getService().acknowledged, ['0/3']);
assert.equal(result.acknowledgedLsn, '0/3');
});
test('transport ACK rejection cannot advance the reported commit LSN', async () => {
const harness = makeCapture(events, { failAcknowledging: '0/3' });
await assert.rejects(harness.run(), { code: 'ACK_FAILED' });
- assert.deepEqual(harness.getService().acknowledged, ['0/1', '0/2']);
+ assert.deepEqual(harness.getService().acknowledged, []);
});
test('callback queue bound refuses overflow without processing subsequent events', async () => {
@@ -184,7 +187,7 @@ test('idle WAL heartbeat completes a fully drained capture after final commit',
const result = await harness.run({ untilLsn: '0/4' });
assert.equal(result.messageCount, 3);
assert.equal(result.changes.at(-1).tag, 'commit');
- assert.equal(harness.getService().acknowledged.join(','), '0/1,0/2,0/3');
+ assert.equal(harness.getService().acknowledged.join(','), '0/3');
assert.equal(result.lastLsn, '0/3');
assert.equal(harness.getService().stopped, true);
});
@@ -195,6 +198,40 @@ test('heartbeat never terminates an open source transaction', async () => {
});
const result = await harness.run({ untilLsn: '0/4' });
assert.deepEqual(result.changes.map(change => change.tag), ['begin','insert','commit']);
- assert.equal(harness.getService().acknowledged.length, 3);
+ assert.equal(harness.getService().acknowledged.length, 1);
assert.equal(result.messageCount, 3);
});
+
+
+test('manual pgoutput capture never enables the driver periodic volatile-WAL ACK timer', async () => {
+ const harness = makeCapture(events);
+ await harness.run();
+ assert.deepEqual(harness.getService().settings.acknowledge, {auto:false, timeoutSeconds:0});
+ const applierCode = readFileSync(new URL('../src/connectors/postgres/cdc-apply.js', import.meta.url),'utf8');
+ assert.ok(applierCode.includes('acknowledge: { auto: false, timeoutSeconds: 0 }'));
+});
+
+test('a complete source transaction is the FIRST WAL position acknowledged', async () => {
+ const harness = makeCapture(events, {commitEndLsn:'0/4'});
+ const result = await harness.run({untilLsn:'0/4'});
+ assert.deepEqual(harness.getService().acknowledged,['0/4']);
+ assert.equal(result.acknowledgedLsn,'0/4');
+});
+
+test('driver refusal to acknowledge a COMMIT fails closed and cannot report success', async () => {
+ const harness = makeCapture(events,{refuseAcknowledging:'0/3'});
+ await assert.rejects(harness.run(), {code:'CDC_ACK_FAILED'});
+ assert.deepEqual(harness.getService().acknowledged,[]);
+});
+
+test('orphan COMMIT is never acknowledged',async () => {
+ const harness=makeCapture([['0/1','commit']]);
+ await assert.rejects(harness.run(),{code:'CDC_PROTOCOL_VIOLATION'});
+ assert.deepEqual(harness.getService().acknowledged,[]);
+});
+
+test('row mutation without source BEGIN is never acknowledged',async () => {
+ const harness=makeCapture([['0/1','insert']]);
+ await assert.rejects(harness.run(),{code:'CDC_PROTOCOL_VIOLATION'});
+ assert.deepEqual(harness.getService().acknowledged,[]);
+});
From 47849fecc61bfa51e1c357f09357ccd1b17a3d88 Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 20:01:25 +0530
Subject: [PATCH 20/21] fix(cdc): enforce mutation effect budgets during hash
and target streaming
---
src/connectors/postgres/cdc-apply.js | 14 ++++++++++++--
src/connectors/postgres/target.js | 9 +++++++++
tests/cdc-apply-unit.test.js | 24 ++++++++++++++++++++++++
3 files changed, 45 insertions(+), 2 deletions(-)
diff --git a/src/connectors/postgres/cdc-apply.js b/src/connectors/postgres/cdc-apply.js
index 981e0efd..4cbb59af 100644
--- a/src/connectors/postgres/cdc-apply.js
+++ b/src/connectors/postgres/cdc-apply.js
@@ -355,10 +355,20 @@ export function createCdcTransactionIdentity({ migrationId, planId, sourceSnapsh
// Incremental ordered-payload hash over a mutation stream: each canonical
// mutation record is length-framed into a running SHA-256, so the hash binds
// exact order without buffering the stream.
-export function hashOrderedMutations(mutations) {
+export function hashOrderedMutations(mutations, { maxMutations = null } = {}) {
+ if (maxMutations != null && (!Number.isSafeInteger(maxMutations) || maxMutations < 1)) {
+ fail('INVALID_CDC_TRANSACTION', 'maxMutations must be a positive safe integer');
+ }
const digest = createHash('sha256');
let count = 0;
const feed = record => {
+ // Enforce the effect budget BEFORE hashing or materializing another
+ // mapped mutation. This is also the FIRST pass of CDC apply: checking
+ // only inside target.commitCdcTransaction is too late for a generator
+ // that exceeds its declared capacity or never terminates.
+ if (maxMutations != null && count >= maxMutations) {
+ fail('CDC_TRANSACTION_BUDGET_EXCEEDED', 'CDC mutation stream exceeded the configured effect budget', { maxMutations, consumed: count });
+ }
const text = canonicalJson(record);
digest.update(`${Buffer.byteLength(text, 'utf8')}:`, 'utf8');
digest.update(text, 'utf8');
@@ -984,7 +994,7 @@ export async function applyCdcTransaction({
}
};
// Pass 1 (streaming, O(1) memory): ordered payload hash + exact count.
- const hashed = await hashOrderedMutations(streamMutations());
+ const hashed = await hashOrderedMutations(streamMutations(), { maxMutations });
const transactionIdentity = createCdcTransactionIdentity({
migrationId: migration.migrationId,
planId: plan.planId,
diff --git a/src/connectors/postgres/target.js b/src/connectors/postgres/target.js
index 5c8bac59..1d75fb48 100644
--- a/src/connectors/postgres/target.js
+++ b/src/connectors/postgres/target.js
@@ -1140,6 +1140,15 @@ export class PostgresTarget {
};
const applyMutation = async mutation => {
+ // An async source can lie about its declared mutationCount. Refuse
+ // the FIRST excess item before executing it, rather than keeping
+ // a database transaction open for an unbounded source iterator.
+ if (appliedMutations >= mutationCount) {
+ await client.query('ROLLBACK');
+ fail('CDC_TRANSACTION_BUDGET_EXCEEDED', 'CDC source emitted more mutations than its declared transaction count', {
+ expected: mutationCount, alreadyApplied: appliedMutations
+ });
+ }
if (!mutation || typeof mutation !== 'object' || Array.isArray(mutation)) {
await client.query('ROLLBACK');
fail('INVALID_CDC_TRANSACTION', 'CDC transaction mutations must be objects', context);
diff --git a/tests/cdc-apply-unit.test.js b/tests/cdc-apply-unit.test.js
index fdc7c806..751e3d58 100644
--- a/tests/cdc-apply-unit.test.js
+++ b/tests/cdc-apply-unit.test.js
@@ -210,3 +210,27 @@ test('reconnect abandons the in-flight transaction and allows clean re-delivery'
await rm(dir, { recursive: true, force: true });
}
});
+
+
+test('bounded ordered CDC hash rejects invalid budgets before consuming a source iterator', async () => {
+ async function *mutations() { yield {op:'upsert',row:{id:1}}; }
+ for (const value of [0,-1,1.5,Infinity]) {
+ await assert.rejects(async () => hashOrderedMutations(mutations(), {maxMutations:value}), /INVALID_CDC_TRANSACTION/);
+ }
+});
+
+test('bounded async CDC mutation hash stops exactly at first excess effect', async () => {
+ const seen=[];
+ async function *mutations() {
+ for(let i=0;i<10;i++) { seen.push(i); yield {op:'upsert',row:{id:i}}; }
+ }
+ await assert.rejects(hashOrderedMutations(mutations(),{maxMutations:3}),/CDC_TRANSACTION_BUDGET_EXCEEDED/);
+ assert.deepEqual(seen,[0,1,2,3]);
+ assert.equal((await hashOrderedMutations((async function*(){yield {op:'upsert',row:{id:1}}})(),{maxMutations:1})).mutationCount,1);
+});
+
+test('bounded synchronous mutation hash refuses an oversized array immediately', () => {
+ const stream=[{op:'upsert',row:{id:1}},{op:'upsert',row:{id:2}}];
+ assert.throws(() => hashOrderedMutations(stream,{maxMutations:1}),/CDC_TRANSACTION_BUDGET_EXCEEDED/);
+ assert.equal(hashOrderedMutations(stream,{maxMutations:2}).mutationCount,2);
+});
From 53773ba511bf2a5ff6a1e5bda802de0e0a36b56c Mon Sep 17 00:00:00 2001
From: dharan <150376358+dharan1007@users.noreply.github.com>
Date: Sun, 27 Sep 2026 20:04:01 +0530
Subject: [PATCH 21/21] fix(cdc): serialize safe standby heartbeat feedback
behind durable commit ACKs
---
src/connectors/postgres/cdc-apply.js | 60 +++++++++++++++++--
tests/cdc-feedback-queue.test.js | 87 ++++++++++++++++++++++++++++
2 files changed, 141 insertions(+), 6 deletions(-)
create mode 100644 tests/cdc-feedback-queue.test.js
diff --git a/src/connectors/postgres/cdc-apply.js b/src/connectors/postgres/cdc-apply.js
index 4cbb59af..7f875462 100644
--- a/src/connectors/postgres/cdc-apply.js
+++ b/src/connectors/postgres/cdc-apply.js
@@ -438,6 +438,42 @@ export async function discoverTargetPrimaryKey({ targetRef, credentialBroker, ex
// durable checkpoint persist -> source acknowledgement.
// Acknowledgement therefore always trails durability; a crash before ack
// replays through ledger reconciliation, a crash after ack is already proven.
+// pg-logical-replication 2.5.x sends periodic feedback from the volatile
+// receive cursor unless the driver timer is disabled. This scheduler sends
+// ONLY an explicitly supplied durable position, serializes heartbeat replies
+// with transaction-commit feedback, and prevents LSN regression under races.
+export function createCdcFeedbackQueue({ acknowledge, initialLsn = null } = {}) {
+ if (typeof acknowledge !== 'function') fail('INVALID_POSTGRES_CDC_CONFIG', 'Acknowledgement sender is required');
+ if (initialLsn != null) requireLsn(initialLsn, 'initialLsn');
+ let queue = Promise.resolve();
+ let lastFeedbackLsn = initialLsn;
+ let closed = false;
+ const send = position => {
+ const job = queue.then(async () => {
+ if (closed) fail('CDC_FEEDBACK_CLOSED', 'CDC acknowledgement queue is already closed');
+ let candidate = typeof position === 'function' ? position() : position;
+ requireLsn(candidate, 'feedbackLsn');
+ if (lastFeedbackLsn && comparePostgresLsn(candidate, lastFeedbackLsn) < 0) {
+ candidate = lastFeedbackLsn;
+ }
+ const accepted = await acknowledge(candidate);
+ if (accepted !== true) fail('CDC_ACK_FAILED', 'Replication driver refused feedback for the durable LSN');
+ lastFeedbackLsn = candidate;
+ return candidate;
+ });
+ // Preserve queue liveness after a refusal, but return the ORIGINAL
+ // rejecting promise so callers cannot silently treat it as success.
+ queue = job.catch(() => {});
+ return job;
+ };
+ return Object.freeze({
+ send,
+ lastSent: () => lastFeedbackLsn,
+ drain: () => queue,
+ close: () => { closed = true; }
+ });
+}
+
function requireSourceTable(value) {
if (!value || typeof value !== 'object' || Array.isArray(value)) fail('INVALID_CDC_APPLY_CONFIG', 'sourceTable is required');
if (typeof value.schema !== 'string' || !value.schema || typeof value.table !== 'string' || !value.table) {
@@ -706,6 +742,9 @@ export function createCdcApplier({
const queue = [];
let streamError = null;
let stopped = false;
+ const feedback = createCdcFeedbackQueue({
+ acknowledge: lsn => service.acknowledge(lsn), initialLsn: fromLsn
+ });
let roomWaiters = [];
let notify;
let notified = new Promise(resolve => { notify = resolve; });
@@ -743,12 +782,19 @@ export function createCdcApplier({
streamError = error;
wake();
});
- service.on('heartbeat', lsn => {
- // Server WAL position with no change attached: the only termination
- // signal on a quiesced source. Tracked for the waterline, and wakes
- // the pump so an idle drain can observe it and stop.
+ service.on('heartbeat', (lsn, _timestamp, shouldRespond) => {
+ // The server can request feedback during an open, uncommitted source
+ // transaction. Respond with the LAST durably saved checkpoint (or
+ // earlier server-ACKed position), never with the heartbeat's volatile
+ // received LSN; queue feedback behind any in-flight commit ACK.
trackPosition(lsn);
wake();
+ if (shouldRespond && !stopped) {
+ void feedback.send(() => serverAck.last ?? fromLsn ?? '0/0').catch(error => {
+ if (!streamError && !stopped) streamError = error;
+ wake();
+ });
+ }
});
service.on('data', async (lsn, message) => {
// Backpressure, never drops: when the queue is full this listener
@@ -771,6 +817,8 @@ export function createCdcApplier({
const stopService = async () => {
stopped = true;
wakeRoom();
+ await feedback.drain();
+ feedback.close();
try { await service.stop(); } catch { /* stopping wins over stop errors */ }
try { await service.destroy(); } catch { /* shutdown must not mask results */ }
};
@@ -779,8 +827,8 @@ export function createCdcApplier({
// names the last SERVER-acked LSN (never the merely applied one).
const flushPendingAck = async () => {
if (serverAck.pending == null) return;
- await service.acknowledge(serverAck.pending);
- serverAck.last = serverAck.pending;
+ const sentPosition = await feedback.send(() => serverAck.pending);
+ serverAck.last = sentPosition;
serverAck.atCount = metrics.appliedTransactions;
serverAck.atMs = Date.now();
serverAck.pending = null;
diff --git a/tests/cdc-feedback-queue.test.js b/tests/cdc-feedback-queue.test.js
new file mode 100644
index 00000000..9fea20d0
--- /dev/null
+++ b/tests/cdc-feedback-queue.test.js
@@ -0,0 +1,87 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { createCdcFeedbackQueue } from '../src/connectors/postgres/cdc-apply.js';
+import { readFileSync } from 'node:fs';
+
+test('queued heartbeat feedback never regresses after a newer committed ACK', async () => {
+ const sent = [];
+ const feedback = createCdcFeedbackQueue({
+ initialLsn: '0/10', acknowledge: async lsn => { sent.push(lsn); return true; }
+ });
+ assert.deepEqual(await Promise.all([feedback.send('0/30'), feedback.send('0/20')]), ['0/30', '0/30']);
+ assert.deepEqual(sent, ['0/30', '0/30']);
+ assert.equal(feedback.lastSent(), '0/30');
+});
+
+test('heartbeat position is resolved only after the preceding ACK finishes', async () => {
+ const sent = [];
+ let release;
+ const gate = new Promise(resolve => { release = resolve; });
+ const feedback = createCdcFeedbackQueue({
+ acknowledge: async lsn => { if (lsn === '0/20') await gate; sent.push(lsn); return true; }
+ });
+ let durable = '0/10';
+ const commit = feedback.send('0/20');
+ const heartbeat = feedback.send(() => durable);
+ durable = '0/30';
+ release();
+ assert.equal(await commit, '0/20');
+ assert.equal(await heartbeat, '0/30');
+ assert.deepEqual(sent, ['0/20', '0/30']);
+});
+
+test('driver refusal fails the caller and never advances the documented ACK cursor', async () => {
+ let refuse = true;
+ const feedback = createCdcFeedbackQueue({
+ initialLsn: '0/10', acknowledge: async () => !refuse
+ });
+ await assert.rejects(feedback.send('0/20'), { code: 'CDC_ACK_FAILED' });
+ assert.equal(feedback.lastSent(), '0/10');
+ refuse = false;
+ assert.equal(await feedback.send('0/15'), '0/15');
+ assert.equal(feedback.lastSent(), '0/15');
+});
+
+test('driver exceptions are not swallowed by queue recovery', async () => {
+ let failNext = true;
+ const feedback = createCdcFeedbackQueue({
+ acknowledge: async () => {
+ if (failNext) { failNext = false; throw Object.assign(new Error('socket closed'), {code:'SOCKET_CLOSED'}); }
+ return true;
+ }
+ });
+ await assert.rejects(feedback.send('0/11'), {code:'SOCKET_CLOSED'});
+ assert.equal(feedback.lastSent(),null);
+ assert.equal(await feedback.send('0/12'),'0/12');
+});
+
+test('feedback rejects malformed positions and permanently closes when requested', async () => {
+ const feedback = createCdcFeedbackQueue({
+ acknowledge: async () => true, initialLsn:'0/0'
+ });
+ await assert.rejects(feedback.send('not-a-position'), {code:'INVALID_CDC_POSITION'});
+ feedback.close();
+ await assert.rejects(feedback.send('0/10'), {code:'CDC_FEEDBACK_CLOSED'});
+ assert.equal(feedback.lastSent(),'0/0');
+});
+
+test('keepalive feedback can only reference a durable cursor, not a volatile receive cursor',async () => {
+ const sent = [];
+ const feedback=createCdcFeedbackQueue({
+ initialLsn:'0/10',acknowledge:async lsn=>{sent.push(lsn);return true;}
+ });
+ const receivedLsn='0/FFFF';
+ const lastPersistedLsn='0/20';
+ await feedback.send(()=>lastPersistedLsn);
+ assert.equal(receivedLsn,'0/FFFF');
+ assert.deepEqual(sent,['0/20']);
+});
+
+
+test('real CDC applier sends all transaction and heartbeat feedback through the durable scheduler', () => {
+ const source=readFileSync(new URL('../src/connectors/postgres/cdc-apply.js',import.meta.url),'utf8');
+ assert.match(source,/const feedback = createCdcFeedbackQueue/);
+ assert.match(source,/serverAck\.last \?\? fromLsn \?\? '0\/0'/);
+ assert.match(source,/const sentPosition = await feedback\.send/);
+ assert.ok(source.includes('auto: false, timeoutSeconds: 0'));
+});