feat: offer PostgreSQL alongside default SQLite with verified backend switching

[n30nex]

Fixes #2144. Related to #2146, #2148 and #2151.

Also fixes #2200, #2212, #2213, #2216, #2217, #2218, #2219, #2221, #2226, #2227, #2228, #2229, #2230, #2231, #2232 and #2233.

SQLite remains the default; PostgreSQL is an explicit installation choice. Updates preserve the recorded backend and database targets. Planned-downtime conversion works in both directions, retains recovery copies and identity sequences, verifies telemetry and initialized accounts, and supports resume/abort. Missing initialized databases cannot become empty replacements.

Current upstream compatibility and upgrades

Merged upstream 6d56f551, retaining its compact observations, on-demand subpaths, companion/device accounts, key backfill and transmission rewrite feed. New queries, migrations, grants and fixtures support both native engines. The ingestor's account projection can read approved channels and linked public keys, without access to private account, session or challenge tables.

Existing PostgreSQL installations have an offline owner upgrade from telemetry schema 1 to 2 and account schema 5 to 6. Bootstrap invokes it before readiness checks and runtime grants complete. It validates every recorded target and its supported physical layout before changing either database, excludes local running services, preserves the selection, and upgrades initialized accounts even when the account UI is disabled. An absent account store stays absent. Each database upgrade is transactional; interrupted work across the two databases is retryable, not a cross-database transaction.

Take native backups of both recorded stores, preserve state/configuration and the previous image, and stop writers before upgrading. Older images may require restoring their matching backups. SQLite account migration recognizes both historical version-6 layouts and preserves accounts, sessions, devices, companion links and identity high-water marks. See operator instructions.

Strict validation of unsupported source objects remains. The hop-index schema proposed in still-open #2214 is not included, and this update does not add an external-snapshot bypass.

Portable fixes from the fork

• Move cold region/area SQL lookups outside the packet-store lock. Refresh immutable neighbor snapshots in the background while retaining the last complete result.
• Reuse bounded integer graph workspaces, defer distance indexing until requested, and prevent requests during chunk loading from consuming the lazy-build gate early. Preserve topology freshness and real readiness responses.
• Guard analytics, Node Analytics and account rendering against late completions, including errors and navigation changes. Handle shared API cleanup rejections and correct regional role counts.
• Keep navigation and resize handles inside their layouts using the stock theme. Preserve All-time packet filters, render safe battery icons, and use current activity evidence in Live details.
• Make pending registration changes atomic. Correct operator update metadata, CRLF initialization and readiness probes.

No new customizer settings or dependencies are introduced. Backend credentials and migration controls remain offline operator settings.

Performance evidence

Original backend comparison and raw results measured c03a258 against upstream 3e21b17 in two independent Linux five-pair studies. Different workloads/runners prevent ranking the candidate backends against each other. PostgreSQL cleanup improved from 11.27 to 0.97 seconds, while many reads slowed and CPU, PSS and storage increased. SQLite's observer-query regression was repaired separately. These historical measurements are not a new benchmark of this merged head.

Bounded startup measurements cover observation-aware memory admission, complete-transmission admission, concurrent live observations and SQL access to history omitted from the cache. The same configured budget may retain a shorter hot window; database retention is unchanged. Disk-planning documentation includes the normalized working snapshot and avoids counting measured WAL twice.

The graph changes carry the paired algorithm benchmark from fork PR #9: on 2,000 nodes/10,000 edges, medians were 10.20 → 1.63 seconds for bridge scoring and 7.02 → 0.75 seconds for coverage scoring. Those measurements used Go 1.25.1 before this upstream merge, not live API traffic. Algorithmic complexity is unchanged; workspace stays O(V+E), with a heap bounded by V. This merged candidate passes independent path-oracle, allocation-bound and readiness regressions. SQL-under-store-lock tests enforce the locking property directly.

Validation

Local validation is used with the contributor's approval because their GitHub Actions minutes are exhausted. Upstream CI is rerunning for this head. The first run exposed an initial-theme timing race in the resize regression test and a missing declaration for an existing shared helper in the lint configuration. Follow-ups 30a812f3 and d132e582 correct these checks. Full frontend lint passes with zero errors, and the complete 24-check chart suite passes again after pushing. Application code is unchanged by these follow-ups. No fresh cloud-CI pass is claimed; earlier CI results apply to earlier commits.

• Full server suites with race detection on SQLite and PostgreSQL 18.6; full ingestor suites on both engines.
• Full migration, user-store and schema suites on both engines, with race detection; configuration/maintenance-lock, mailer and signature-validation suites also pass.
• All standalone frontend suites and operator shell regressions pass. Unchanged Linux service-user tests: 103/103.
• Local Chromium: core runner 157/160, with 3 reported fixture-dependent skips; focused analytics, navigation, charts, tables, icons and account journeys: 204/204.
• ARM64 packaged image builds successfully. Isolated Compose checks pass for fresh SQLite/PostgreSQL startup, SQLite recreation, recorded PostgreSQL v1/v5 upgrades with disabled accounts, and telemetry-only upgrades. Selection bytes, telemetry, authentication fields, identity sequences and grants are preserved. Direct and proxied readiness pass; owned test resources are removed. No MQTT publish was included in this packaged check.

The current live Canadaverse deployment is separate from this PR update.

[n30nex]

Review feedback addressed (commit 7f655ae)

1. The deployed ingestor can inspect packets_v; the actual-role startup regression passes.
2. Restore checks now change only database URL paths, preserving credentials and connection settings.
3. Blacklist QA uses native PostgreSQL binding: 93 SQL assertions and 14 connection-safety tests pass.
4. The benchmark can run through an explicit PR opt-in with exact source checks and read-only permissions.

Independent follow-up reviews are clear. Full CI and the paired performance results remain pending; this PR stays in draft.

[n30nex]

Review feedback addressed (commit b5941a8)

1. Benchmark setup now enables I/O accounting and verifies every required counter before building. Sampler failures retain safe diagnostics without masking workload errors; 22 controls pass.
2. The scope-icon E2E waits for initial theme refresh and the intended node render. All existing assertions remain; 20 repeated runs and an independent run passed.

The previous run passed Go, race, and packaged deployment checks. A fresh full run and benchmark pilot are starting; comparative performance remains unverified.

[n30nex]

Review feedback addressed (commit e07bf46)

1. The isolated benchmark now checks PostgreSQL startup before building, uses loopback TCP explicitly, and preserves safe startup diagnostics.
2. WebSocket latency excludes warmup while retaining full-stream delivery checks, negative lags and sample-count limits.

All 28 controls pass, and independent follow-up review is clear. The prior candidate passed every standard CI check, including native backup restoration and packaged startup. Its benchmark failed before PostgreSQL measurements; the new run must complete before any comparison is claimed.

[n30nex]

Review feedback addressed (commits 8e7a11b, caac986)

1. Imports now preserve SQLite date text exactly through copying and resume verification. Populated dropped-packet fixtures cover fractional precision, offsets, nulls and invalid numeric storage.
2. Benchmark artifacts retain sanitized errors from the shipping importer's actual log prefix.

The full local S import passed: 30,000 transmissions, 90,000 observations and all 18 table digests match; the source is unchanged. The full importer suite and 29 benchmark controls pass. Independent review is clear. Standard CI passed on the previous candidate; the performance comparison is restarting with this fix.

[n30nex]

Review feedback addressed (commit 7b2c19a)

1. Removed synthetic observer/path collisions without changing row counts or missing-observer coverage.
2. Both engines use a 2 GiB store and 3 GiB server limit within the unchanged 3 CPU/6 GiB stack budget.
3. Generator regressions now execute before measurement.

Go 1.27.2 full/hot startup retained all 112,000 transmissions and 1,792,000 observations for 65 seconds with zero evictions. Independent review and 31 harness controls pass. The small paired pilot completed earlier; the five-pair Linux comparison is restarting. No speedup is claimed.

[n30nex]

Review feedback addressed (commit c315d7d)

The 100-event/s SQLite qualification filled its 1,024-event input queue during a retention pass: 82 schedules were dropped, while durable effects matched completed writes. That failed result remains linked in the benchmark documentation.

The next experiment uses 50 events/s and 20 HTTP requests/s for both engines. Queue size, five pairs, durations, corpus, durability, resource limits and zero-loss gates are unchanged. It is labeled separately from the failed 100/s qualification.

All 33 harness controls and workflow lint pass. Linux must still qualify the new rate; no speedup is claimed.

[n30nex]

Review feedback addressed (commit ce023e0, including 1c0281a and 0a24721)

1. Resume now handles interrupted two-store cutovers without rewriting completed data or resetting another finalizer's readiness.
2. Upgrade instructions preserve credentials, configuration and rollback files; packaged checks exercise the default HTTP listener.
3. Coverage queries avoid the measured generic-plan regression while preserving results.
4. Memory-peak reset and reads now share their required file descriptor.

Full local server/migrator suites, 195 frontend suites and 36 benchmark controls pass. Independent reviews are clear. The earlier five-pair comparison passed data checks; final-candidate CI and remeasurement are pending. This remains draft until results are published.

[n30nex]

Review feedback addressed (commit fde34bf)

1. Published the five-pair upstream comparison, complete results, source hashes and raw-artifact links. Runtime and benchmark sources are byte-identical to measured ce023e0.
2. Reported gains alongside slower writes/queries, increased CPU/PSS/storage, and the observed memory limit. No capacity or blanket speedup claim.
3. Added a five-step operator guide covering recovery copies, import, resume, validation and rollback.

The measured candidate passed all standard CI, packaged startup, browser validation and independent review. The final documentation update is going through CI; the completed benchmark is not being repeated.

[n30nex]

Scope update: SQLite will remain the default, with PostgreSQL offered explicitly at installation or update. Updates must preserve the configured backend; switching requires verified offline migration of telemetry and accounts.

I’ve returned this PR to draft while adding backend selection and coverage for both configurations. The existing performance report measures the PostgreSQL-only candidate; it is not evidence for the forthcoming optional-backend implementation.

The separate SQLite benchmark, retention and memory work under #2146 continues alongside this, without duplicating #2147.

[n30nex]

Review feedback addressed (commit 2408dcf2)

1. Merged upstream 6d56f551 and carried over the portable fixes from the merged Canadaverse fork PRs: store-lock handling, graph allocations and startup work, neighbor freshness, stale frontend rendering, account registration, operator readiness, and display/layout corrections. The upstream merge and logical fixes have separate commits.
2. Added the owner-only upgrade path required by existing PostgreSQL installations. It checks every recorded store before schema changes, excludes running local services, preserves disabled accounts, and keeps absent accounts absent. Native tests and isolated packaged v1/v5 upgrades verify retained data, session/authentication fields, identity counters, selection bytes and runtime grants.
3. Updated operator guidance for backups, retry/rollback, migration disk headroom and the observation-aware cache. Choose the database for the workload and available hardware: SQLite stays the default, and PostgreSQL has additional service, memory, storage and backup costs. Historical performance reports remain pinned to their measured revisions; this update does not claim a new end-to-end backend speed comparison.
4. Full local SQLite/PostgreSQL server, ingestor, migration, account and schema suites pass, including race checks where applicable. The frontend suite, 103 Linux service-user checks, ARM64 image build and isolated packaged startup/upgrade checks pass. Chromium reports 157/160 core checks with 3 fixture-dependent skips, plus 204/204 focused checks. Local validation was completed before pushing with the contributor's approval while their Actions minutes are exhausted. Upstream CI started automatically for this new head and is still running.

The pending #2214 schema remains outside the supported import layout, and no external-snapshot bypass was added. The ongoing soak on 01b2a941 should remain identified with that older head when comparing results. The live development instance was not redeployed by this PR update.

[n30nex]

Review feedback addressed (commit 30a812f3)

1. Fixed the resize regression test exposed by the first upstream CI run on both database backends. Initial theme loading could replace the table between mousedown and mouseup. The test now waits for the actual initial theme event and an initialized, visible table before dragging.
2. Kept the real behavior checks: handles remain anchored, there is no page overflow, dragging grows the column by more than 20 pixels, widths are saved, and sticky headers remain sticky. Added checks for nonempty handles and stable table identity during the drag.
3. The complete chart suite passes 24/24 locally before and after the push. This follow-up changes only the test; the verified application code and database migration are unchanged. The new upstream CI run is pending.

[n30nex]

Review feedback addressed (commit d132e582)

1. Declared the existing shared getEffectiveHeardMs helper as read-only in the frontend lint configuration. Its definition and script load order were verified; no lint rule was disabled and no runtime code changed.
2. Full frontend lint now passes with zero errors. The chart browser suite also passes 24/24 again after this push. This completes the two fixes identified by the first upstream CI run: initial-theme settlement in the resize test and the missing shared-helper lint declaration.

The current application source is identical to the locally verified database/migration/package candidate. Upstream CI is rerunning on the latest head; no fresh cloud pass is claimed yet.

[nullrouten0]

Soak moved to d132e582

The soak above ran on 01b2a941, which predates upstream #2147 and the other read-path lock fixes now merged into this branch. We stopped both stacks at 2026-10-15 23:33Z and upgraded them in place to d132e582, keeping the ~18 h of data (about 600k observations each) so the two sides stay comparable. Results from 01b2a941 stay pinned to that head. The PostgreSQL v1→v2 owner upgrade ran automatically in bootstrap against that real database: rows intact, telemetry schema version 2, ready. (Bootstrap prints [migrate] unsupported telemetry schema version 1 (binary supports 2) before upgrading, which reads like a failure on a run that succeeded.) We also lowered retention.packetDays from 14 to 3 so the daily prune starts deleting real volume within three days.

On the earlier points

Thanks for the sizing example in docs/storage.md; that answers the headroom question.

The import point is still open. validateSource refuses any trigger, and any table that is not in the dbschema manifest, and the docs now also say not to delete objects to make validation pass. #2214 adds two tables (observation_hops, observation_hops_coverage) and three triggers on observations, so an install that has run #2214 has no supported path to PostgreSQL, and if #2214 lands first that is every upgraded install. Either of these would close it: (a) a way to mark known derived objects, tables and triggers alike, as skipped on import (adding the tables to the manifest would instead copy a SQLite-only index into PostgreSQL); or (b) a documented step that drops those five objects before import. The index they maintain is derived entirely from observations.path_json, so dropping them loses no data. Tell us which you'd accept and we'll match it from the #2214 side.

On the cache change: the docs/storage.md paragraph helps people choosing a backend, but the operators it affects are existing SQLite installs that won't open that page. One startup log line when observation-aware admission keeps fewer packets than the old per-packet estimate would (both counts and the resulting hot window) would make it visible where they look.

The last update also brings in about twenty UI, analytics and operator fixes from another fork. Splitting those into their own PRs would make this one easier to review, and would let soak results be attributed to the storage change rather than to everything that moved with it.

Edited 2026-10-15: corrected option (a). The importer also refuses #2214's two tables, not just its triggers.