Repository navigation
Brand the Canadaverse CoreScope deployment - #1
Merged
Merged
Conversation
…pa-clawbot#1966) Eviction removes raw wire hops but leaves resolved full-key entries in `byPathHop`, retaining expired transmissions and stale relay counts/scopes. Filter every hop bucket once per eviction batch using the existing evicted-ID set, remove duplicate references and empty buckets, and clear discarded pointer slots. Bulk relay aggregation now owns its bucket snapshots before releasing the read lock, so eviction and raw-path updates cannot mutate an in-flight reader. Three existing handler test fixtures also wait for index readiness or explicitly simulate not-ready state, preserving their original 200/503 assertions. Fixes Kpa-clawbot#1908. Validation: - Regression commits fail before their corresponding fixes: resolved keys/counts/scopes remain after eviction, and saved relay snapshots change during eviction. - Targeted eviction, relay, scope, cache and concurrent-reader checks pass under `-race`; coverage includes time/cap eviction, missing resolved-path prefetch, disabled membership indexing, duplicate references, retained backing arrays and surviving entries. - Local browser smoke: nodes, node details/path attribution and analytics render using the fixture-backed Go server. - The last full Windows server race run, before the final snapshot-copy correction, had one remaining DB-only timing failure (`TestGetChannelMessagesPerfLargeChannel`: 2.198s against a 1.5s budget). The final correction was checked with focused race tests. The unchanged ingestor suite also cannot create one symlink without Windows privileges. These thresholds/assertions were preserved; full Linux Go/E2E results still require upstream CI approval. Performance tradeoff: cleanup is O(total indexed pointers) per nonempty eviction batch, under the existing write lock. The minute-based ticker pays for one sweep instead of repeated scans of shared raw buckets. No per-transmission string index or dependency is added. Synthetic benchmark medians (three single-iteration runs, shared Windows host): | Transmissions | Evicted | Before | After | |---:|---:|---:|---:| | 30,000 | 1 | 1.07 ms | 14.19 ms | | 30,000 | 3,000 | 56.27 ms | 61.68 ms | | 30,000 | 7,500 | 83.58 ms | 65.54 ms | | 100,000 | 1 | 0.30 ms | 56.95 ms | | 100,000 | 10,000 | 949.93 ms | 190.71 ms | | 100,000 | 25,000 | 1,834.48 ms | 320.38 ms | Fixture: eight raw plus eight resolved hops per transmission, two observations, 2,048 relays; 480,000/1,600,000 hop entries. Timing includes acquiring the store lock and omits unrelated secondary indexes. Small batches now pay for the complete sweep; shared-host timing is noisy. Owning the bulk reader's arrays also has a measured cost on cold/bulk recomputation, rather than cached hits. Snapshot medians from three samples of ten iterations: | Transmissions / relay nodes | Before time / bytes per operation | After time / bytes per operation | |---|---:|---:| | 30,000 / 50 | 0.0068 ms / 5,416 B | 23.65 ms / 4,101,435 B | | 30,000 / 2,000 | 0.1374 ms / 196,768 B | 26.77 ms / 4,274,336 B | | 100,000 / 2,000 | 0.1376 ms / 196,768 B | 27.89 ms / 13,959,337 B | These are total snapshot costs, comparing the unsafe header-only snapshot with owned pointer arrays. Cleanup guarantees here apply to `byPathHop`; other indexes and existing periodic bulk-cache freshness are outside this change. Following Kpa-clawbot#1922, this runtime fix is separate from the release-routing and frontend-runner PRs. Current Go and E2E job results should be assessed separately from workflow-approval or staging-runner state.
Red commit: `5a5ecb6` (local browser: 14 passed, 8 behavior assertion failures before the fix). Hash Issues links now restore `bytes=1|2|3` for the selected control and its matrix/collision data. Missing or malformed values default to one byte. Selector clicks, section/top links, tab-bar changes, filters and theme refreshes retain the chosen view through the existing URL helper. Fixes Kpa-clawbot#1914. - E2E assertion added: `test-issue-1306-collisions-terminology-e2e.js:242`. The existing CI-selected harness passes 23 checks, including distinct nonempty collision rows for each byte size. Its original assertions remain. - Browser verified: `http://127.0.0.1:55634` with the local fixture API, plus reviewed matrix/risk screenshots. Region refresh passed; area coverage skips because the fixture has no areas. - Required frontend checks pass: 99 filter, 18 aging, 666 helpers; URL helpers pass 18. Three independent reviews found no blocking issues; their coverage suggestion is included in `fb482fe`. - Added work parses URL state and updates six links. Rendering and bulk requests are reused; no backend, configuration, dependency or CI-list changes. - A broader smoke run timed out at Live autocomplete (Kpa-clawbot#1110); full-suite success is not established. ## Preflight overrides - The external `run-all.sh` is absent. Corresponding scope, PII, syntax, whitespace and CSS checks passed; no SQL, migration or image changes require those gates. - Red browser evidence is local. Upstream CI execution remains a separate approval gate, as discussed in Kpa-clawbot#1922.
Red commit: `2bf9be8` (local Chromium: 3 passed, 3 intended assertion failures). CI: [run](https://github.com/Kpa-clawbot/CoreScope/actions/runs/33988112224) awaits maintainer approval (`action_required`); 0 jobs started. Remove the unqualified aggregate Avg SNR row from node side-panel Overview and full-detail stats, following option 3 in Kpa-clawbot#1149. Heard By retains each observer's SNR reading. Fixes Kpa-clawbot#1149. - E2E assertion added: `test-issue-1281-location-row-e2e.js:224`. Three new browser cases cover desktop side/full and mobile full views with a numeric aggregate and distinct positive/negative observer readings. Existing packet-location assertions remain intact. - Browser verified: local Chromium; 6 cases passed after push. Screenshots: `coverage/issue-1149/issue-1149-desktop-side-panel.png`, `coverage/issue-1149/issue-1149-desktop-full-detail.png`, and `coverage/issue-1149/issue-1149-mobile-full-detail.png`. - Validation: packet filter 99/99, aging 18/18, frontend helpers 666/666; XSS, CSS-variable, syntax, whitespace and PII checks passed. - Independent reviews: adversarial, lifecycle expert and TDD reviewers found no required changes. One initial browser navigation timed out; the unchanged parent rerun passed 6/6. - Performance/config: two production row deletions; no new requests, loops, timers, settings or customizer implications. Backend unchanged; Go suites were not rerun. Fix commit: `d7c68f3`. ## Preflight overrides - External `run-all.sh` is unavailable on this host. Scoped branch, red/green, PII, CSS, XSS and whitespace checks were run directly. The diff adds no migrations, SQL attribution or image markup.
…t#1968) Red commit: `0988bc0` (local browser: 3 passed, 8 behavior assertion failures before the fix). The selected node now has a compact outline in long “Paths Through This Node” chains, in the side panel and full detail page. Matching uses complete public keys without case sensitivity; same-prefix siblings and unresolved hops stay unmarked. Existing links, escaped names, warnings and ambiguity underlines are preserved. Fixes Kpa-clawbot#1153. Its prerequisite Kpa-clawbot#1144 is already merged. - E2E assertion added: `test-issue-1146-path-link-contrast-e2e.js:220`. The existing CI-selected harness passes 11 checks across 18-hop paths, both themes, desktop/mobile, and the renderer fallback. Review follow-up `bd8118f` verifies the marked ambiguous hop's dashed underline. - Browser verified: `http://127.0.0.1:55635`; desktop/mobile path screenshots were inspected. The broader smoke runner exited successfully with fixture-dependent skips. - Required frontend checks pass: 99 filter, 18 aging, 666 helpers. CSS variables, seven CSS self-tests, 31 XSS sink checks, 17 XSS gate self-tests and XSS diff preflight pass. - Three independent reviews found no blocking issues. Traversal remains linear with no new requests, settings, dependencies or cache invalidation; styling uses the existing customizer token. ## Preflight overrides - The external preflight runner is absent; corresponding scoped gates passed. Red browser evidence is local, with upstream CI approval tracked separately under the process in Kpa-clawbot#1922. - Existing rapid-navigation map resize timer errors remain visible in browser logs and are outside this change.
Red commit: `2f1a50e` (local Chromium: 2 passed, 7 assertion failures). Ownership regression: `059ab49` (9 passed, 4 assertion failures). CI: [run](https://github.com/Kpa-clawbot/CoreScope/actions/runs/33988139479) awaits maintainer approval (`action_required`); 0 jobs started. Rapid navigation or closing node detail could leave a delayed resize targeting a removed or replacement map. Disposal now cancels its timer, each resize captures its own map, and delayed responses respect the view owning the current map. Stale side-pane responses are ignored before rendering. Fixes Kpa-clawbot#1972. - E2E assertion added: `test-issue-1206-resize-observer-leak-e2e.js:216` and `:292`. This existing CI-selected suite covers navigation, replacement deadlines, close/Escape, no-location rendering, and late error/success responses. Existing observer-growth assertions remain intact; readiness waits replace fixed sleeps. - Browser verified: local fixture with real Chromium and Leaflet; 13 browser checks passed after push. Evidence: `data/node-map-validation/post-push-browser.log` and `data/node-map-validation/evidence.md`. - Validation: frontend unit suites 99/18/666 passed; JavaScript syntax, CSS variables, whitespace, PII and XSS checks passed. Backend unchanged; Go suites not rerun. - Independent adversarial, expert and TDD reviews found no required changes. Two original navigation checks initially timed out; the unchanged parent rerun passed 13/13 (`parent-browser-confirm.log`). - Performance/config: one timer handle, no packet/node loops or new requests. Tests enforce zero stale invalidations and one resize at the surviving map's deadline. Existing 100ms delay retained; no new settings or throughput claim. Fix commits: `2492a65`, `1dc090d`. ## Preflight overrides - External `run-all.sh` is unavailable. Scoped branch, red/green, PII, CSS, XSS and whitespace checks were run directly; no migrations, SQL attribution or image markup are added.
… evidence (Kpa-clawbot#1971) Carries SaarMesh-Bot's implementation from the closed Kpa-clawbot#1867 forward onto current master, 67 commits later, and surfaces the result on the per-node Reach report. The declared region list a repeater answers with is now stored on the node as configured_scope, normalised to the same leading-# syntax default_scope already uses so the two are directly comparable. That normalisation is the point @cwichura raised on Kpa-clawbot#1865 and @dborup agreed with before the original PR closed. Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
…ade (Kpa-clawbot#1974) Closes Kpa-clawbot#1794. Follow-up to Kpa-clawbot#1793, decided **before** the upgrade because the handshake is the resource being protected. - Deny list of addresses and CIDRs → 403 - Per-IP concurrent connection cap → 403 - Per-IP upgrade rate limit over a rolling minute → **429**, not 403: a temporary refusal should not read as "never come back" - Rejection counters split by cause in `/api/stats` under `websocket` ### The decision this feature lives or dies on Most CoreScope installs sit behind nginx, Caddy, Traefik or an ingress. `cdn_detection.go` says so in as many words: it deliberately excludes `X-Forwarded-For` from its CDN signals precisely because *every* reverse-proxied install sets it. For those deployments `r.RemoteAddr` is the proxy, `127.0.0.1` for every visitor on earth. A per-IP cap keyed on that address protects nobody and hands the sixth legitimate browser tab a 403. That is a self-inflicted outage wearing the costume of hardening. So: - **`X-Forwarded-For` is believed only from an address listed in `webSocket.trustedProxies`.** From anywhere else it is attacker-supplied, and trusting it would let anyone mint a fresh source IP per connection, which is strictly worse than having no limit at all. - **When the peer looks like a local reverse proxy and no `trustedProxies` is set, the per-IP limits are skipped**, and one warning names the setting that fixes it. Silently refusing real users is the worse failure. - **The deny list still applies there**, because it is the operator's explicit instruction rather than an inference. That is the answer to @mcode6726's question on the thread: it is neither "always the socket address" nor "always the header", and the operator decides which by naming their proxy. ### Two deliberate departures from the issue body **`maxConnsPerIP` ships as 0 (off), not 5.** Carrier-grade NAT puts thousands of unrelated mobile subscribers behind a single public IPv4. A cap of 5 refuses real visitors on phones while a scraper simply rents more addresses: all of the cost, none of the benefit. `upgradesPerMinPerIP` ships at **30 and on**, because that one *is* safe under CGNAT: a real client upgrades a handful of times per minute even while reconnecting, so 30 leaves ordinary traffic untouched while flattening a reconnect loop. A pointer type distinguishes "unset" from an explicit `0` that turns it off. **The default deny list is not shipped.** The thread proposed seeding 44 CIDRs for one VPS provider after a single scraper was seen at `23.111.177.6`. I have left it out: blanket-blocking a hosting provider by default breaks legitimate operators who host there, is undiscoverable by the person locked out (they see a bare 403), and ages badly as ranges get reassigned. The mechanism is here and `config.example.json` shows exactly how to configure it, so any operator who wants that list can have it in one line. If you want it shipped as a default anyway, that is your call as maintainer and it is a one-line change. ### Verification 19 tests, including all five the issue specifies as TDD requirements, each marked with the issue's own wording. Beyond those five: - a **bare address** in the deny list works, not just CIDR form. Operators write `1.2.3.4`, and silently ignoring that would be the worst possible failure for a deny list: it looks configured and blocks nothing - an unparseable deny entry is skipped and logged, not fatal. One typo must not take the server down - one client behind a trusted proxy does **not** exhaust another client's budget behind the same proxy, which is the entire point of honouring XFF - changing a forged XFF from an untrusted peer buys no fresh budget - `release` frees a slot and is **idempotent**, because `Unregister` can run twice for one client and double-crediting would leak slots - a **rejected** upgrade does not consume rate budget, or a retrying client could never recover once its window cleared - limits skipped for loopback and private peers; deny list applies anyway - a nil limiter allows everything, so a `Hub` built without `ConfigureLimits` behaves exactly as before - idle per-IP state is collected, while a record with a live connection never is Full `cmd/server` suite green, `gofmt` clean. ### Not done - No runtime config reload; restart required. Listed as optional in the issue. - No `WS_DENY_IPS` env override. Also listed as optional. - From the OWASP expansion in the first comment: `maxPayload` and the idle/read timeout are **already in master** (`SetReadLimit`, `SetReadDeadline`). The ping/pong heartbeat is not, and is not in this PR either; it is a separate change to the read/write pumps and belongs in its own review. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…clawbot#1964) Successful release fast paths publish image tags but never dispatch the job that creates the GitHub release and decrypt binaries. Dispatch `deploy.yml` from both image routes. A default-off `images_published` input skips E2E and image rebuilding only for an already-published tag; missing or mismatched images keep the complete fallback without requiring new inputs on older workflow definitions. Go validation still gates the release binaries, checkout and version flags retain the tagged source, and the existing release action uploads both architectures before publication. Missing binary files now fail publication. Fixes Kpa-clawbot#1956. Validation: - `node test-issue-1956-release-routing.js` executes the actual workflow shell steps with registry and dispatch commands stubbed. Covers matching, missing and mismatched images; failed retag and Go validation; branch/PR boundaries; and both tagged binary commands. - The original test commit fails because a matching image dispatches zero artifact workflows; the fix passes the same assertion. - Existing release workflow Go checks, decrypt/channel tests, YAML parsing and actionlint pass. - Both static Linux amd64 and arm64 binaries cross-build with verified architecture and version metadata. Actual registry publication and GitHub release creation were not exercised. Existing immutable releases and old tags that contain older workflow definitions are outside this fix. Following Kpa-clawbot#1922, this is a focused release-routing PR. A separate repair for Kpa-clawbot#1858 rewrites the shared frontend test runner; merging this first lets that repair retain this regression in its authoritative list. Please assess current Go and E2E job results separately from workflow-approval or staging-runner state.
Kpa-clawbot#1973) Closes Kpa-clawbot#1845 for the question in its title. The alerting ask that came later in the thread is deliberately **not** in here; see the bottom. ### The gap @Jonher937 asked to flag repeaters that stopped communicating for x days, to find remote gear that has died. Today the Nodes page has Active/Stale with thresholds fixed at 72h for infra and 24h for everything else, plus a Last Heard filter that selects nodes heard **within** a window. Neither answers "show me what has been quiet for over a week". ### What this adds A `Silent for` select beside Last Heard: 1d, 3d, 7d, 14d, 30d, each labelled with the number of nodes it would select. ``` [ All ] [ Active ] [ Stale ] Last Heard: Any v Silent for: over 7d (23) v ``` Counts are computed **before** the silence filter is applied, so the dropdown keeps showing what the other windows would select instead of collapsing to the one already chosen. The choice persists in `localStorage` like the neighbouring filters and is mirrored into the URL as `?silent=7d`, so the view can be pasted to whoever owns the silent gear. The URL sync is wrapped in try/catch, because it is a convenience and must never stop the filter working. ### The part worth reviewing: one definition of freshness, not two `getNodeStatus` has been relay-aware since Kpa-clawbot#1598, while `nodes.js` separately computed `statusAge` from the ADVERT timestamp alone. Filtering on the latter would have listed a repeater as silent for ten days while its own badge on the same row said active, and it would have done so for **exactly** the nodes Kpa-clawbot#1598 exists to protect. So the freshness rule is extracted into `window.getEffectiveHeardMs` in `roles.js`, and `getNodeStatus` now calls it. Behaviour is unchanged, there is now one source. Reviewers should look hardest at that refactor rather than at the select. A node never heard from at all scores `Infinity`, so it matches every window instead of silently dropping out of the filter. ### Why the thresholds are fixed values and not derived I measured the alternative before writing this, on a 1179-repeater mesh, and posted it on Kpa-clawbot#1611: replacing a fixed threshold with `3 x per-node advert median` fixes 8 false "silent" flags and newly mis-flags **28 currently-active nodes**, because a 3h-median node gets a 9h threshold. Raising the global default to 144h rescues 7 and hides 27 genuinely dead repeaters. Both are net-negative. A user-chosen window sidesteps the whole question: the operator picks what "too long" means for their mesh, which is what @Jonher937 asked for in the first place. ### Verification - `test-frontend-helpers.js`: **666 → 680 passed, 0 failed**. Fourteen cases covering `NaN` rather than `0` when nothing is known (0 is a real timestamp and would sort as very old rather than unknown), the full `_liveSeen > _lastHeard > last_heard > last_seen` precedence, a recent relay beating a stale advert, a stale relay **not** dragging a fresh advert backwards, relay alone sufficing, `room` counting as infra while `companion` does not (a `last_relayed` on a companion is meaningless and must not rescue it), case-insensitive roles, the legacy `(role, ms)` call shape, and the 72h boundary asserted at 71h and 73h. - One of those failed on first run and **the test was wrong, not the code**: `9e7` ms is 25h, which is correctly active for infra. Fixed, and the boundary is now asserted explicitly so nobody repeats it. - `eslint` on the changed files: 0 errors. The warnings present are the same ones master already reports. - No server change, no new API, no new column. No cache-buster bump needed: `__BUST__` is substituted at startup in `cmd/server/main.go:570`. ### Not done **No alerting.** @fokcuk asked on the thread for notification when a repeater they look after goes silent, which is a different product: subscriber identity, an evaluation loop and delivery, none of which exist today. That deserves its own issue and a design call rather than being shimmed into `nodes.js`, which is also what the triage concluded. This PR gives the operator the view; it does not push to them. Sizes and boundaries (1d/3d/7d/14d/30d) are a judgement call. Say the word if a different set fits real operator habits better. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ed scopes (Kpa-clawbot#1976) One row per repeater whose configured region list is known, answering a question no other view answers: you declare these regions, but were you seen forwarding them? default_scope says what a node's adverts were observed under and transported_scopes (Kpa-clawbot#1751) says what it carried, but nothing lined the declared list up against observed forwarding. The declared side merges every confirmed-scope source the database carries, newest answer per node wins, rather than naming one. On a stock install only nodes.configured_scope (Kpa-clawbot#1865/Kpa-clawbot#1971) exists and it degrades to the one-source case; deployments that collect the same fact another way keep working. Reading a single hard-coded source would have rendered an empty page on the very instance the evidence came from. Declared and observed are compared through normScope, so a leading "#" and a bare region name are one region. Unobserved regions render neutral, not red: absence over a short window is weak evidence, which the page header already states in words. Ported from a long-running fork deployment with its 17 server tests, rewired to the upstream data source, plus 14 frontend cases asserting rendered markup.
## Summary - Bumps the Go toolchain used to build/test to 1.27: `golang:1.27-alpine` in `Dockerfile` and `Dockerfile.go`, and `go-version: '1.27'` in the three `actions/setup-go` steps in `.github/workflows/deploy.yml`. - Each module's `go.mod` `go` directive is intentionally left at `1.22` — no 1.27-only language features are being adopted, and a 1.27 toolchain builds a `go 1.22`-declared module without issue. ## Test plan - [x] `go build ./...` + `go vet ./...` for all 13 modules (`cmd/server`, `cmd/ingestor`, `cmd/migrate`, `cmd/decrypt`, 10 `internal/*` packages) under Go 1.27.0 - [x] `go test ./...` passes for `cmd/server`, `cmd/ingestor`, `cmd/migrate`, `cmd/decrypt` - [ ] `docker build` against the new `golang:1.27-alpine` base (Docker wasn't available in the sandbox this change was prepared in — needs a check in CI or locally)
…ing it (Kpa-clawbot#1982) Closes Kpa-clawbot#1977. Supersedes Kpa-clawbot#1952. Follow-up filed as Kpa-clawbot#1983. ## What §10.2 did ```bash q="SELECT COUNT(*) FROM transmissions WHERE from_node = '$TEST_PUBKEY';" qq=$(printf %q "$q") if ! count=$(ssh_t "docker exec … sqlite3 … $qq" 2>/dev/null); then count=$(ssh_t "sqlite3 … $qq" 2>/dev/null || echo "") fi ``` The injection is not reachable today — `TEST_PUBKEY` is hex-gated and the script `exit 2`s before the SQL is built. The problem is that the SQL layer's safety rests entirely on that outer gate rather than on the SQL layer itself. Kpa-clawbot#1952 proposed doubling embedded quotes; that is string escaping, not parameterisation, which is why it was withdrawn in favour of this. ## What this does Per the four points in the sign-off on Kpa-clawbot#1977: **1. Bind the value.** A constant `SELECT` and a bound `:pubkey`, fed to sqlite3 on stdin. The SQL no longer crosses the remote shell as a command word, so there is no `printf %q` on the query at all any more. **Why hex rather than `.parameter set :pk '<value>'`.** Dot-command arguments are split on whitespace, so a payload containing a space produces too many arguments — and sqlite3 responds by printing the `.parameter` help to **stdout**, exiting **0**, and leaving `:pk` **unbound**. `COUNT(*)` then returns 0, which reads exactly like a passing security fix. `-bail` does not catch it. Verified on 3.51.0: ``` $ printf ".parameter set :pk '' OR 1=1 --'\nSELECT COUNT(*) FROM transmissions WHERE from_node = :pk;\n" \ | sqlite3 -bail ptest.db .parameter CMD ... Manage SQL parameter bindings # <- help, on stdout … 0 # <- :pk never bound $ echo $? 0 ``` `.parameter set :pk 1+1` also binds the integer `2` — the value is evaluated as an SQL expression and only falls back to a text literal when evaluation fails. So interpolating into the `.parameter set` line trades one hazard for another. Hex-encoding removes the quoting layer instead of adding one: the value is bound as `cast(x'<hex>' as text)`, so its contribution to the SQL text is drawn from the alphabet `[0-9a-f]` only. Nothing to quote, no tokenizer arity hazard, and it holds for **arbitrary** input rather than only for hex-gated input — which is the point. Verified against a fixture table holding two rows, one of them `deadbeef`: | value | result | exit | |---|---|---| | `deadbeef`, bound as `cast(x'6465616462656566' as text)` | `1` | 0 | | `' OR 1=1 --`, bound the same way | `0` | 0 | | `' OR 1=1 --`, interpolated the current way | `2` (whole table) | 0 | | query against a DB with no `transmissions` table, `-bail` | `Parse error … no such table` on **stderr** | **1** | **2. Probe the capability, not a version.** `resolve_sqlite_runner` binds `corescope-probe-ok` and asserts it comes back — a round trip, not a bare `.parameter init`, so the positive control runs against the operator's actual binary rather than one we pin. If neither the container nor the host qualifies, it fails loudly and names what is needed: ``` ❌ retain-failed: no sqlite3 able to bind a parameter on the target tried: docker exec -i corescope-prod sqlite3, then sqlite3 on runner@example need: the sqlite3 CLI reachable over ssh, supporting '.parameter set' OCI runtime exec failed: exec: "sqlite3": executable file not found in $PATH bash: line 1: sqlite3: command not found ``` There is deliberately **no** interpolating fallback. That would leave the vulnerable path in place under a nicer name. **3. The hex gate is kept**, with its comment updated to say why: for the SQL layer it is now defence in depth rather than the only guard. Redundant is not the same as wrong. **4. The exit status and stderr survive.** `-batch -bail -init /dev/null -noheader -list` (stop at the first SQL error; ignore the operator's `~/.sqliterc`, where a stray `.mode` would make the count unparseable; stdout is exactly the number). Query stderr is captured and printed on failure rather than sent to `/dev/null`, so a broken query is distinguishable from a legitimately empty result. Probe stderr is collected too, and printed only if *both* probes fail — the container miss is the known-normal case, so surfacing it on every run would be noise. ## Also fixed An existing double-count in §10.2: the `TARGET_DB_PATH unset` branch incremented `$fails` and then left `count=""`, so the generic branch incremented it a **second** time for the same failure. `read_retain_count` now gives §10.2 exactly one increment point. Opportunistic cleanup in a file already being touched (AGENTS.md line 318). ## Tests New `qa/scripts/test-blacklist-sql.sh`, wired into the `go-test` job. 24 assertions, modelled on `scripts/staging/test-disk-monitor.sh`. Both directions are asserted, because a zero from a command that failed proves nothing: - **Positive control** — `deadbeef` still returns its row (`1`, exit 0), and so does `cafebabe`; an absent pubkey returns `0`. - **Negative** — `' OR 1=1 --` returns `0` while the table demonstrably holds 2 rows, and the old interpolated form is asserted to leak all `2`. That last assertion is what makes the `0` above worth something. - **Error surfacing** — the same SQL against a DB with no `transmissions` table exits non-zero with a message on stderr and nothing on stdout. - **Alphabet** — `sql_hex_literal` output matches `^x'[0-9a-f]*'$` for the SQL payloads, a backslash, `$(id)` / backticks, an embedded newline, `héllo`, and a 4096-byte repetitive string. That last one is a regression guard for `od -v`: without the flag `od` collapses repeated identical lines to `*`. - `run_sqlite` with no resolved runner refuses rather than guessing. Group 2 skips loudly (rather than silently) if `sqlite3` is not on PATH; group 1 needs no sqlite3 and always runs. **Mutation-tested** — each of these breaks the suite, so the assertions have teeth: | mutation | caught by | |---|---| | restore full interpolation | `injection payload → 0 rows — expected '0' got '2'` | | naive `.parameter set '%s'` | `expected '0' got '.parameter CMD ...'` | | drop `od -v` | alphabet failure on `*`, plus `expected '8192' got '33'` | Commit 1 is a behaviour-neutral refactor that moves the imperative body into `main()` behind a `BASH_SOURCE` guard, so the test can source the script and exercise individual helpers. Same idiom as `scripts/staging/disk-monitor.sh:99`. ## Verification - `bash qa/scripts/test-blacklist-sql.sh` → 24 passed, 0 failed - `bash -n` on both scripts - All three runtime paths exercised end to end with PATH shims for `ssh`/`docker`/`sqlite3` against a real fixture DB: success (`sqlite3 runner: host`, count 2), query failure (classified message + `Parse error … no such table`, `fails=1`), and no-capability (the loud block above, both probe stderrs, `fails=1` — not 2) - The new step lands inside `go-test`, which runs when `changes.outputs.code == 'true'`; `qa/scripts/*.sh` does not match that job's `^docs/|[.]md$|^LICENSE$` documentation filter, so it is not skipped ## Deliberately out of scope - **The `docker exec` branch is dead on current images** → filed as Kpa-clawbot#1983. The app container has no `sqlite3` at all: `Dockerfile:15` is pure-Go SQLite with no CGO, and the `apk add` installs only `mosquitto mosquitto-clients supervisor caddy wget`. So the host "fallback" is the only path that has ever executed, silently, because both branches discarded stderr. This change keeps both branches and merely makes the outcome visible (`sqlite3 runner: …` on every run). - **`-readonly` on the target DB.** Tempting, and verified compatible with `.parameter` (the binding table lives in the TEMP database), but a WAL database needing journal recovery can refuse a read-only open. Adding it here risks exactly the "trades an unreachable injection for a script that does not run" outcome flagged in the Kpa-clawbot#1952 thread. Worth its own issue. - **The other `2>/dev/null` sites** in this file, which also sit awkwardly with `qa/README.md`'s "Don't silence stderr". Only the §10.2 lines named in the sign-off are touched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…wider scan (Kpa-clawbot#1986) The Scope Audit credits a transmission to `path[last]` only. On a flood-family route every forwarder appends its own hash to the END of the path (`internal/packetpath/route.go`), so `path[last]` does not mean "forwarded this packet", it means "was the transmission an uplinked observer heard directly". Every earlier hop forwarded the same packet and is discarded. The last-hop rule is genuinely required for DIRECT routes, which consume hops from the front, so their `path[last]` is the route’s far end rather than the transmitter. But `scopeAuditForwarderScanQuery` already restricts to `route_type IN (0, 1)` via `scopeConformanceForwarderRouteTypesSQL`, where that hazard cannot arise, so inside this query the restriction only throws evidence away. ## What it costs the page today Measured on a live-shaped instance, 206 declared repeaters, 965k transmissions, 7d window: | | before | after | |---|---|---| | repeaters with no attributable evidence of any kind | 133 of 205 (65%) | 30 of 206 (15%) | On a 1000-packet flood sample the mean path length is 7.08 hops, so the last-hop rule keeps 394 of 2789 hop observations (14%), and 85% of the nodes seen forwarding never appear as a last hop at all. Those repeaters have every region they declare reported as "declared, not observed", which is the page presenting a gap in our own attribution as a finding about someone else’s repeater. ## Rule 0: what widening it costs, and what pays for it Reading every hop multiplies the rows the scan returns: a 7d window yields **3,470,188 hop rows** from 1,368,761 observations carrying a path. Cold cost before this change was 16.7s for 7d and 4.0s for 24h, of which SQLite accounts for 2.7s. The rest was the Go side reading rows. Three changes, in order of what they bought: 1. **The scan carried `scope_name` and `first_seen` on every hop row.** Both are columns of `transmissions`, and at 43 hop rows per transmission the same two values were re-read that many times. They now come from one query over the same window keyed by transmission id, both inside one read transaction so a transmission arriving between them cannot appear in the hop scan with no metadata to attribute it by. The hop scan carries two columns instead of four. 2. **The hop is lower-cased into a stack buffer** instead of through `strings.ToLower`. 1,026,814 of the 1,284,897 hops in a 24h window are stored uppercase, because `packetpath.DecodePathFromRawHex` writes them that way, and the great majority match no declared target, so that allocation was paid millions of times to answer "no". The `(target, txID)` de-duplication key became a struct for the same reason. 3. **The compute ran outside the cache mutex**, so every request arriving on a cold window ran its own full scan concurrently. It now sits behind a singleflight, the same treatment `/api/observers` and `/api/nodes/{pubkey}/reach` already have, and the 7d window gets a 5 minute TTL while 1h and 24h keep 30s. At 30s a single reader with 7d open keeps the instance recomputing more than half the time, for an aggregate that moves at the pace of a week of traffic. Result, warm process: | window | before | after | |---|---|---| | 1h | 0.155s | 0.140s | | 24h | 4.04s | 2.79-2.89s across six samples | | 7d | 16.7s | 11.6s | | repeat inside TTL | ~1ms | ~1ms | **Rejected alternatives, measured on the same database**, so the next reader does not have to re-derive them: | approach | rows returned | time in SQLite | |---|---|---| | the query as written | 3,470,188 | 2.7s | | pre-filter on the declared targets’ first 4 hex chars | 1,971,126 | 20.9s | | `GROUP BY t.id, hop` | 965,025 | 38.0s | | `SELECT DISTINCT t.id, path_json` | 1,229,966 | 17.7s | The query plan is already index-driven (`idx_transmissions_first_seen`, then `idx_observations_tx_ts`), so there is no missing index behind this: the rows are inherent to the data. Note for anyone attempting a hop comparison in SQL: a case-sensitive comparison silently drops most attributable hops, per the 80% figure above. ## Tests - a mid-path hop is attributed (the case behind the 65% blind spot) - a DIRECT transmission whose `path[last]` **is** the target is still not attributed. With the last-hop rule gone this is the only thing standing between the audit and misattribution, so it gets its own test rather than relying on the route filter being obvious - one transmission counted once per target even when it appears on several hops of the same path, which the `(target, txID)` de-duplication now carries alone - a hop longer than the 4-char floor resolved by its own length, which nothing pinned before: every other test seeds 4-char hops - the per-window TTL, so collapsing it back to one constant has to delete the reason - a second request inside the TTL served from cache rather than recomputed. The cache path had no test at all `cd cmd/server && go test ./...` passes (168s), `go vet` and `gofmt -l` clean. Server-side only, no API shape change, no frontend change. Browser validation: run against a live instance carrying this change, the Scope Audit renders 220 rows matching the API row for row, and the per-node scopes page still answers with its route-type mix.
… name (Kpa-clawbot#1987) Follow-up to Kpa-clawbot#1986, and the second half of the same problem. `ScopeAuditForwarding` drops rows whose `scope_name` is the empty string with a bare `continue`. That empty string is the ingestor's "transport-scoped, but no configured region key matched `code1`" state (`scopeNameForDB`), so those packets name no region and can never satisfy a declared one. The consequence is on the page: **a repeater forwarding a region this instance holds no `hashRegions` key for is reported exactly like a repeater forwarding nothing at all.** The audit presents a gap in the reader's own configuration as a finding about someone else's hardware. This counts them per target, exposes the count as `observedUnmatchedPackets`, and renders it as a caveat chip beside the scope chips. ## Why it is not a rare edge Measured on a live instance before this landed: of 613 `notObserved` entries across 205 repeaters, **260 named a region that never appeared under any name in the whole 7-day window**. Two of them (`behss`, `fm-112`) were hash-verified as genuinely forwarded traffic the instance simply could not name: packet `0a065d41d51f1f77` decodes to `code1=9209`, which is exactly the code `#fm-112` derives over that packet's own payload. That instance had 16 region keys configured against 124 distinct region names its repeaters declare. A stock install has fewer. ## What the counter is not It is deliberately **not** folded into `unscopedPackets`. The two are opposites: | | meaning | what governs it | |---|---|---| | `unscopedPackets` | the packet carried no scope at all (`scope_name` SQL NULL) | the `*` wildcard | | `observedUnmatchedPackets` | the packet IS scoped, this instance holds no key for that region | nothing the repeater declares | For the same reason the new count never feeds `wildcardContradiction`, which counts only plain unscoped floods. `scopeNameForDB` in the ingestor is the source of truth for that three-state encoding, and the comments point there rather than restating it. It is also distinct from `ambiguousHops`, and the distinction is the point of the chip: that one is a pubkey-prefix collision between two repeaters and is nobody's fault, this one is a missing entry in the reader's own configuration and they can act on it. Saying which is which is what stops someone investigating an innocent repeater. ## Frontend The chip reuses the muted dashed treatment of `.sa-chip-ambiguous` on purpose: both are caveats on the row's finding rather than findings themselves, and neither may compete visually with the red/green scope chips beside them. It renders nothing for a non-numeric count. The value is server-supplied, and a truthiness check would put the literal string `NaN forwarded packets` on the page if that ever stopped holding. ## Docs `docs/api-spec.md` had **no entry for `GET /api/scope-audit` at all**, so this adds one: query parameter, full response shape, and the notes a client needs (the three traps the per-node endpoint documents apply here identically, `*` is never a scope, and "never asked" is not "declared nothing"). The new field is documented there rather than in isolation. ## Tests - the counter on a last-hop and on a mid-path hop - an unmatched packet enters neither `agg.scopes` nor `unscopedPackets`, which is the confusion this field exists to prevent - the field on the API row - six frontend cases: zero renders nothing, a missing field renders nothing (older server), the chip carries its count and class, singular and plural are both grammatical, the title names the cause and the fix, and a non-numeric count renders nothing rather than `NaN` `cd cmd/server && go test ./...` passes, frontend 712 assertions pass, `go vet` and `gofmt -l` clean. Rule 0: the counter is one increment on a branch that already existed as a `continue`, inside a loop this PR does not change. No new query, no new pass over the data.
…pa-clawbot#1988) `matchScope` returns the first configured region whose derived code equals the packet's `code1` and stops there. Go randomises map iteration order per `range`, so when two configured regions collide on a payload, the stored region name depends on which key the runtime happened to visit first. **The same packet can be named differently on two runs of the same binary**, and neither answer is evidence of anything. ## How often this actually happens `code1` is two bytes, so any two configured regions collide on a given payload with probability 1/65536. That is a curiosity at 5 configured regions and routine at 150. Measured on a live instance carrying 159 configured regions, over 41.7 hours and 147,535 transport-scoped packets: **374 collisions, 0.253% of decisions.** They concentrate on four key pairs rather than scattering, because `code1` is an HMAC over the payload: a payload that collides collides every time it is seen, and a flooded packet is seen by many observers. The existing comment sizes the function for "≤ 50 regions". A live BE/NL instance declares 126 distinct region names across its repeaters, so operators are already past that. ## The rule `matchingRegions` returns every match; `matchScope` applies one rule: - exactly one match names the packet - several matches name nothing The candidates are equally sourced, there is no principled winner between them, and storing a wrong region name is worse than storing none. `""` is already the ingestor's "transport-scoped but unnameable" state (`scopeNameForDB`), so an ambiguous packet lands in a state the rest of the system already understands rather than in a new one. Nothing downstream needs to learn a new value. The collision is logged, because it is otherwise invisible: such a packet is stored exactly like one whose region this instance holds no key for. An operator watching an unnameable count grow deserves to see which of their own configured regions are colliding, since the fix is theirs to make. ## Rule 0 Cost is unchanged: the same single pass over the same keys, it just no longer stops early. The early exit was worth nothing on the common path, where zero or one key matches and the loop runs to the end either way. Worst case is unchanged at one HMAC per key per transport-scoped packet. There is no indexable shortcut to reach for. `code1` is an HMAC over the packet payload, so nothing is payload-independent to index on, and the old comment suggesting a "pre-indexed lookup table" is removed rather than left as a false lead for the next reader. ## Tests Three, and the fixture matters: - an unambiguous packet still gets its region name - a genuinely colliding payload stores the unmatched state instead of a coin flip - the ambiguous case run 50 times, because a first-match matcher passes a single iteration roughly half the time The collision is **found by searching payloads** (~65k tries, fractions of a second) rather than asserting on a hand-picked `code1`. The case only exists when the matcher genuinely finds two names for one packet, and a fabricated code would only prove the test agrees with itself. `cd cmd/ingestor && go test ./...` passes apart from `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which needs `SeCreateSymbolicLinkPrivilege` and fails on Windows on master too. `go vet` and `gofmt -l` clean.
…pa-clawbot#1989) Follow-up to Kpa-clawbot#1988, which made an ambiguous match deterministic. This adds the second tier of keys that ambiguity rule was needed for. A transport-scoped packet can only be named by a region key this instance holds. `hashRegions` is a hand-maintained list, so every region a node forwards that nobody typed into the config is stored unmatched, and everything downstream reports that region as **absent** rather than as **unnameable**. The instance already knows the names, though: `nodes.configured_scope` holds what the observer `/neighbors` ingestion (Kpa-clawbot#1865) confirmed each node is configured for. This derives keys from those names, on top of the explicit list rather than instead of it. **Default off.** An absent config block leaves behaviour byte-for-byte unchanged, asserted by tests rather than argued. ## Sources Two, mirroring what the server's `AllCurrentDeclaredRegions` already merges, so the derived tier sees exactly what the Scope Audit sees: | source | availability | |---|---| | `nodes.configured_scope` | always — the column is part of the schema, written by the `/neighbors` path | | `node_declared_regions` | optional, where a deployment fills it by other means | The optional table is probed via `sqlite_master` before it is read. A stock install does not have it, and its absence must not abort a refresh the first source could answer on its own. ## The two spellings The sources spell the same region differently, and both are accepted: - `configured_scope` carries the leading `#` that `normalizeScopeList` adds, because every other stored scope value has one - an OTA answer in the optional table carries the bare name - `loadRegionKeys` already prefixes a missing `#` before hashing So `regionNameAcceptable` canonicalises before judging, and hands back the bare name the caller re-prefixes. What it rejects is what cannot be a region name at all: `*` (the flood wildcard, not a region), a comma (it would split the name on the next round-trip through a comma-separated column), a second `#`, whitespace, non-ASCII, NUL padding from a stale client, and anything past 32 characters. The rules are deliberately structural rather than about meaning. A real declared set contains entries that look like junk, but a blocklist on string values is unmaintainable, and the cost of one bad name is a single slot out of the cap plus a 1-in-65536 collision chance. `*` is skipped in **two** places on purpose: in the filter, so no key is ever derived for it, and in the source count, because nearly every node declares it and counting it would overstate both the cap arithmetic and the refresh log on every deployment. ## Rule 0 Each derived key costs one HMAC per transport-scoped packet and raises the random 2-byte collision rate by 1/65536. So: - the tier is **capped** (default 256) - over the cap, names are kept by how many distinct nodes declare them, so a one-off local name is dropped before a region half the network uses - benchmarked linear at ~0.65µs per key: at 314 keys that is 217µs per packet, 0.0008% of one core at the 0.037 transport-scoped packets/s this network produces There is no indexable shortcut to reach for, for the same reason as in Kpa-clawbot#1988: `code1` is an HMAC over the payload, so nothing is payload-independent. The set is an `atomic.Pointer` to an immutable snapshot. A refresh builds the replacement off to the side and swaps the pointer, so the ingest path never blocks on a rebuild. `refreshDerived` is a load-then-store rather than a CAS loop, which is safe only because exactly one goroutine calls it: the refresh ticker, plus one synchronous call at startup. The comment says so, rather than leaving the type looking as though it tolerates concurrent refreshers. ## Tier 2, and the counters When several keys match one packet and exactly one of them is explicit operator config, the explicit one wins: an operator who typed a region into `hashRegions` outranks a name overheard on the air. Ambiguity between two equally-sourced keys still stores unmatched, unchanged from Kpa-clawbot#1988. Counters tally how each packet was decided (unique / explicit-over-derived / ambiguous / none) and are logged on the refresh tick. They exist to answer one question with data rather than estimation: whether a third tier that breaks ties on path evidence is worth building at all. Measured on a live instance at 159 keys over 41.7 hours and 147,535 packets: **0.253% ambiguous**, with explicit-over-derived at zero because that instance has exactly one derived key. ## Rule 8 `maxDerived` and `refreshMinutes` are configurable values and belong in the customizer eventually. Documented in `config.example.json` for now, flagged here so it is tracked rather than forgotten. ## Tests - default off, and a refresh that is a no-op while disabled - the name filter across both spellings, the wildcard, and each structural rejection - ranking by declarer count, then recency, then name, so the result is deterministic rather than churning between refreshes - `configured_scope` as a source, including that a node counts once per name and the newest answer wins - both sources merged, neither dropped - the explicit-over-derived tie-break - the derived tier replaced rather than merged on refresh, so a region that stops being declared leaves the key set and the cap keeps meaning something `cd cmd/ingestor && go test ./...` passes apart from `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which needs `SeCreateSymbolicLinkPrivilege` and fails on Windows on master too. `go vet` and `gofmt -l` clean, `config.example.json` still parses.
…n traffic (Kpa-clawbot#1990) Follow-up to Kpa-clawbot#1987, and the point of counting that traffic in the first place. A region this instance holds no `hashRegions` key for is **unnameable, not absent**. Kpa-clawbot#1987 says so with a caveat chip. This settles it wherever the evidence allows: derive `SHA256("#region")[:16]` from the repeater's own declaration and HMAC that repeater's own unmatched packets with it. Same computation the ingestor performs at ingest, with the candidate set narrowed from every configured key to this repeater's handful of declarations. Where it fires, a grey "declared but not observed" chip becomes a green one and the caveat count shrinks by the packets it explained. ## Two packets, not one `code1` is two bytes, so an unrelated name matches a given packet with probability 1/65536. Across ~400 unmatched packets and ~124 declared names, chance alone produces roughly one false match per refresh. Two matches on the same region for the same repeater is (1/65536)², about one in four billion. Lowering the threshold to one would not make this noisy, it would make it **unsound**, so the constant carries that arithmetic and a test rather than a comment. A region with exactly one hit stays grey and reports its single hit, so the page can say why it is still shown as not observed instead of leaving the reader to wonder. ## What it deliberately does not do **It writes nothing.** Read-time only. A wrong answer expires with the window instead of sitting in `transmissions.scope_name` until someone runs a repair, and `cmd/server` stays read-only per the invariant in AGENTS.md. **`notObserved` remains the single source of chip colour.** `regionEvidence` says only HOW a region was established. Two fields that can disagree about the same fact is how this column got confusing in the first place. ## Rule 0, including the part that was wrong at first The naive shape is `targets × names × packets` HMACs: 205 × 124 × 400 ≈ 10M. Caching per `(region, transmission)` pair cuts the HMACs to ~50k. **That measured 501ms**, because the HMACs had become a rounding error while the *iteration* stayed cubic at 10.2M map lookups. Re-keyed per region, holding the set of matching transmissions, it is **36ms** at the same worst-case shape: a region is HMACed over every packet once, and a target then asks one question per declared region instead of one per (region, packet). Most declared regions match nothing, so the common case is a single map lookup and no packet loop at all. `hmacCount` exists so a test can assert the first mistake cannot come back; the benchmark exists because only it caught the second. ## Both axes are bounded, because neither is bounded by the data The "~400 packets in a 7 day window" this was sized against is a property of one instance's configuration, not of the feature: the ingestor stores the unnameable state for every transport-scoped packet no configured key names, so an instance with few or no `hashRegions` entries — the stock state, and the one this helps most — has **every** scoped packet in that set. - the window query takes the 4096 most recent candidates and reports truncation, which the handler logs, so a grey chip on a sampled refresh is not read as "not forwarded" - the declared list is capped at 32 names per repeater: it arrives from a collector that validates each entry's shape but never how many entries there are - measured at the cap: **306ms** for 205 targets over 124 names, against 29ms for the shape a real network produces Because both caps make the evidence a sample, the response carries `observedUnmatchedSampled`. Without it a client subtracts a capped numerator from an uncapped total and overstates the unexplained traffic with no way to know it is doing so. The chip subtracts only evidence for regions **absent** from `notObserved` — a single-hit region the server refused to accept is not called explained either — and says "at most N" when the count was sampled. ## Verified on live data Six repeaters clear the threshold in a 7d window on a real instance. One of them: `nl-nb` green with 3 corroborating packets and the tooltip stating the count, `belml` still grey on 1, and the caveat chip reading 31 of 34 packets unexplained rather than 30. ## Tests `scope_verify_test.go` covers the HMAC-input walk against a real transport-flood packet captured from a live instance (a hand-built fixture would only prove the parser agrees with itself), that `regionCode` does not fold case, the threshold in both directions, the memo's HMAC count, both bounds with their truncation flag, and the benchmark at cap size. Handler-level tests cover a region verified into green, a single hit left grey with its count reported, and the sample-size field. `cd cmd/server && go test ./...` passes (77s), frontend 723 assertions pass, `go vet` and `gofmt -l` clean.
…pa-clawbot#1991) Closes Kpa-clawbot#1980. `app.js` opens a WebSocket on every page load and fans messages out through `onWS()`/`offWS()`. `live.js` ignored that channel and opened its own socket to the same endpoint. `Hub.Broadcast` does no per-client filtering, so both carried the identical full packet stream and **every viewer on the live map pulled it twice**. That page is the one people leave open for hours, so it was a standing multiplier on origin bandwidth rather than a burst. ## Measured, before and after Against a running instance, leaving the live map and returning while counting WebSocket constructions: | | new sockets on re-entry | constructed by | |---|---|---| | before | 1 | `at connectWS (live.js:3315)` | | after | 0 | nothing | And with one viewer on the live map, the server now reports **one** WebSocket client for that viewer, with the live feed counter climbing normally (7 to 31 over one navigation cycle, 71 on a fresh load). ## The change The live map subscribes to the shared channel like every other view, and unsubscribes in `destroy()` rather than closing a socket the rest of the app still needs. `connectWS()` drops any existing registration before adding a fresh one. An early return looks like the natural guard against double subscription, but it keeps the previous visit's closure registered, and re-registering without dropping the old one renders every packet twice. Dropping first is idempotent either way and always binds the current page. Reconnection becomes `app.js`'s business, since it owns the socket. That moved `WS_RECONNECT_MS` out of the only place that honoured it, so `app.js` now uses it too. It comes from `roles.js` and operators set it as `wsReconnectMs`; after this change it applies to the one socket everyone shares, or to nothing at all. ## Tests Three, in the sandbox that already loads `live.js` with a Leaflet mock: - the page registers exactly one listener on the shared channel and constructs no WebSocket of its own - re-entering leaves exactly one listener, and it is the new one rather than the previous visit's - the handler ignores messages that carry no packet `node test-frontend-helpers.js` passes (726 assertions), `node test-packet-filter.js` passes. ## One note on the history The second commit on this branch claims the early-return guard broke rendering on re-entry, citing a real measurement. The measurement happened, the attribution was wrong: the zero counter came from the WebSocket constructor patch I had installed to count sockets, which interfered with the page it was measuring. The third commit records that rather than rewriting it away. The change is kept because dropping the old registration first is the clearer contract, not because the guard was broken. ## Rule 0 Strictly less work than before: one socket per viewer instead of two, one JSON parse instead of two per packet, and no second reconnect loop. Nothing is added to the hot path; a listener already existed for every other view.
…t proved less than they claimed (Kpa-clawbot#1993) Three leftovers from reviewing the scope-audit series (Kpa-clawbot#1986, Kpa-clawbot#1987, Kpa-clawbot#1990). None is urgent; all three are the kind of thing that gets harder to explain the longer it sits. ## `scopeHMACInputs` accepted a payload `DecodePacket` rejects Its comment says it walks the same offsets as the decoder, and it does, minus the `maxPacketPayload` bound the decoder enforces. Unreachable in practice: such a packet never reaches the database with an empty `scope_name` in the first place, so the verifier never sees one. Worth closing anyway, because the comment claims the two agree. A verifier that accepts what the decoder refuses is a small divergence today and an hour of confusion on the day it matters. ## The corroboration test seeded the same packet twice The threshold of two rests on `code1` being two bytes: one match is 1/65536 by chance, two on the same region is (1/65536)². That argument needs two **independent** observations. The test seeded `realTransportFloodPacket` twice. Identical payloads derive identical codes, so it was one observation counted twice, and it would have passed just as happily against an implementation that counted rows rather than deriving anything. It now seeds the real captured packet plus a second one built for a different payload, both deriving to `#fm-112` on their own. **The feature was never wrong here.** `transmissions.hash` is unique and `ComputeContentHash` is path-independent, so two rows always mean two distinct payloads in production. Only the test failed to demonstrate the property it is named for. ## Naming at ingest and verifying at read time had no test together They were built separately, in Kpa-clawbot#1990 and Kpa-clawbot#1989, and the interaction between them is not exotic: with derived region keys enabled, a packet that used to be stored unnameable now arrives with a name. The audit must then report that region as observed by the ordinary route: - present in `agg.scopes` - absent from `notObserved` - **not** claimed by `regionEvidence`, which exists to explain regions that could only be established by verification Getting that wrong is quiet. The chip stays green while the reason underneath it is wrong, and a reader asking "how do we know this" gets the wrong story. ## Verification `cd cmd/server && go test ./...` passes (254s), `go vet` and `gofmt -l` clean. No production behaviour changes beyond the payload bound, which rejects input that cannot occur.
…it matters (Kpa-clawbot#1994) `go test -race ./...` on `cmd/ingestor` reports **six data races** on master. None is in production logic. All six come from test helpers that outlive the test that started them, which is why the detector blames whichever test happens to be running: two different tests failed on two consecutive runs of the same code. ## What was racing **`StartStatsFileWriter` had no way to stop.** Two tests start it at a 50ms interval, and its goroutine then runs for the rest of the process. It reads the package-level `readProcSelfIOFn` hook, which a later test replaces to inject a fake, so the write and the read race. The same leak explains the stray log lines about writing stats into temp directories that were already cleaned up. It now returns a stop function that closes the goroutine and waits for it to exit. Production ignores the return value and runs for the process lifetime exactly as before; the two tests call it through `t.Cleanup`. **The migration test read a log buffer while a goroutine wrote to it.** `log.Logger` serialises its own writes, but `logContains` read `buf.String()` outside that lock while `RunAsyncMigration` kept logging after the call that started it had returned. The capture helper now uses a mutex-protected buffer. That one is worth calling a real race rather than a test artefact: a concurrent read during a buffer grow can panic outright with "concurrent map read and map write"-class behaviour, not merely trip `-race`. ## Measured `go test -race ./...` on linux/arm64 under go1.27.1: **exit 0, zero races, 704s**. ## The CI job, and why it is shaped this way The server has had `-race` since Kpa-clawbot#1208. This closes the same gap for the ingestor, which carries an `atomic.Pointer` snapshot (the region key set from Kpa-clawbot#1989) whose safety has been an argument rather than a measurement. Two deliberate choices, because a check that costs too much gets switched off: - **Its own job, not a step inside "Go Build & Test".** Appending `-race` there puts its ten-odd minutes on the critical path, taking the pipeline from roughly 20 minutes to roughly 32. As a separate job it runs beside the E2E job (15-17 minutes) and hides inside that window. - **Only when `cmd/ingestor/**.go` changed**, decided by the existing change-scope job, which already gates the heavy jobs on documentation-only PRs. A frontend or docs PR cannot introduce a data race in the ingestor. Pushes to master always run it, as they already do for `code`. Nothing `needs:` the new job. Adding it to `build-and-publish` would mean a skipped race job skips everything downstream, which is the opposite of what a conditional check should do. It reports as its own check; whether that blocks a merge is a repository setting rather than workflow logic. ## Scope Test helpers, one production signature (`StartStatsFileWriter` now returns a stop function), and the workflow. No change to what the ingestor does at runtime.
…ot#2002) ## What `scopeMatchCounters` (unique / explicit-over-derived / ambiguous / none) lives only in the ingestor process, and the only way to read it is the periodic log line. A restart zeroes the counters, and recreating the container removes that log with it, so a measurement in progress cannot be recovered afterwards. That happened here on 2026-09-10: a 24h ambiguity measurement completed, two deploys followed before it was read, and nothing on disk held the number. `/var/lib/docker/containers/*/*-json.log` had no earlier copy. The tally gates a real decision (whether the collision tie-break from the `autoRegionKeys` design is worth building), and that needs days of traffic, so it has to survive the process counting it. ## How A single-row table, `scope_match_totals`: - `OpenStore` restores the counters from it, so counting continues instead of restarting. - The 5-minute stats ticker writes them back, and so does the shutdown path, which is what a deploy triggers. - `since_unix` carries the window start across restarts, so the ratio keeps a denominator. The periodic log line now prints it. Saving rides the **stats** ticker, not the region-refresh ticker: matches are recorded for every transport-scoped packet, including on instances that never enable `autoRegionKeys` and so never start the refresh loop. The table is **not** in `internal/dbschema` on purpose. The server neither reads nor PRAGMA-detects it; putting it in `AssertReady` would make an older DB fail the server's startup check over data the server never looks at. A failed restore is logged and ingestion continues. Losing an observability counter is not a reason to stop ingesting. ## Tests Four, all in `cmd/ingestor/scope_match_tally_test.go`: - totals and `since_unix` restored across a close/reopen - a fresh DB gets its anchor row immediately, so the first window has a start time - recording after a restore adds to the carried total instead of counting from zero - repeated saves keep exactly one row Full `cmd/ingestor` suite green locally except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails on Windows only (`os.Symlink` needs a privilege this account lacks) and predates this branch. `go vet` and `gofmt` clean. The race detector was not run locally (no cgo toolchain here); the `race-test` job covers it, since this branch touches `cmd/ingestor`. ## Not done No UI or API surface for the tally. It is still read from the log line or straight from the table.
…ng it to stop (Kpa-clawbot#2003) Verified before merging: on upstream/master `go test ./cmd/ingestor -run TestMQTTStallWatchdog -count=20` fails; on this branch the same command passes. The flake blocked CI on Kpa-clawbot#2000.
…stalling ingest (Kpa-clawbot#2000) Reviewed at ecf0b37: query plans dumped and confirmed index-driven for all three statements, termination proven against concurrent ingest (first_seen is always time.Now()), FK child-first ordering required and correct, writer-stats assertions non-racy. Two low findings noted on the PR for follow-up: the dropped RowsAffected error now gates the loop, and ~0.53s batches will trip defaultSlowWriterMs=500.
…pa-clawbot#2006) Closes Kpa-clawbot#2001. Two review rounds plus a re-review; findings and evidence on the PR. Verified on a deployment against live data: the field over 1653 nodes, marker tints per filter state, the marker title and popup Scope row reaching the DOM, and the colorblind-preset cascade. The audit and the map are held to the same classification by an end-to-end test that fails when either side's wildcard handling drifts.
…Kpa-clawbot#2005) Fixes Kpa-clawbot#2004. Two review rounds; findings and evidence on the PR. Verified by injecting the stylesheet into a running deployment and reading computed styles before and after: the badges gain background, size, uppercase and padding; at 430px the Config column stays visible; the sorted column keeps its accent. The new test fails on the eight unstyled classes against the pre-fix tree, on .ns-truncated against the first fix, and on a re-added column-hiding rule.
Verified: every value is SHA-256(name)[:16] per internal/channel.DeriveKey, 319/320 exact (Public is the fixed firmware default), no duplicate hashes, file parses at 320 entries.
…pa-clawbot#2010) Reproduced on a live instance: 41px column for a 53px button at 1440px, 29px at 1024px, max-width:0 and overflow:hidden on the cell. Injecting the .col-action rule took it to 65px with nothing cropped. The new test fails 13 of 16 checks when only the CSS block is removed.
…clawbot#2009) Verified against production: ingestor writes enc_<HH> (db.go:2282), store.go:3536 filters on it, live values are uppercase two-digit hex with thousands of packets (enc_28: 2708), /api/channels omits them, and /api/packets?channel=enc_28 returns rows. New test passes at 14 and fails when the dedupe guard is disabled.
…ot#1965) Make `test-all.sh` the authoritative standalone frontend runner for npm and CI. Restore stale assertions and reject missing, duplicate, removed or undocumented inventory entries. Fixes Kpa-clawbot#1858. Rebased onto `a2f039d4`. Retains release-routing, map scope-state and Scope Audit stylesheet tests, adds `test-packets-local-channels.js` to the sorted runner, and classifies `test-neighbor-map-btn-clip-e2e.js` under browser. Its separate CI browser step is preserved. Inventory: 280 root suites, 167 standalone, 113 requiring separate setup. The icon repair fixes two suites red on master: `test-issue-1648-m2-emoji-scan.js` and `test-issue-1648-m6-final-sweep.js`. Node/live configured-scope confirmations now use the existing accessible Phosphor check sprite. Values, visibility conditions and scanner assertions are preserved. - Red evidence: `89e45a9` inventory assertions; `4e255df` accessible confirmation assertion. The latest rebase also reproduced both unclassified-file failures before adding their entries. This follow-up only changes runner/classification configuration and counts; no test files modified. - Local validation: all 167 standalone suites; 27 M2 browser checks and 16 neighbor geometry checks in Chromium. Syntax, whitespace, PII, CSS-variable and XSS checks passed. - Browser coverage includes populated/empty/null configured scopes in node and live views. - No new dependencies, requests, application settings or Go changes. Workflow outside the unit step matches master. - Windows validation uses process-local UTF-8 settings. Encoding and node-reach confirmation follow-ups remain separate, as requested. ## Preflight override External `run-all.sh` is unavailable; applicable repository checks were run directly.
…2013) (Kpa-clawbot#2016) ## What `buildMQTTOpts` (`cmd/ingestor/main.go:591`) never called `SetClientID`, so with paho.mqtt.golang v1.5.0 every ingestor connected with a zero-length ClientID and `CleanSession=true`. The session identity then depended on the broker. This PR: - adds an optional `clientId` per `mqttSources` entry (`cmd/ingestor/config.go:29`) - when unset, uses `corescope-<name>-<6 hex chars>` (`cmd/ingestor/main.go:653`). The name is reduced to `[0-9A-Za-z-]`, with the broker host as fallback when the name is empty. The suffix comes from `crypto/rand` and changes on every ingestor start. - sets the ID once per source in `buildMQTTOpts` (`cmd/ingestor/main.go:616`). paho copies the options into the client and reuses them for every reconnect, and the watchdog force-reconnect reuses the same client, so the ID is stable for the life of the process. - logs the ID on connect: `MQTT [tag] connected to <broker> as client <id>` (`cmd/ingestor/main.go:150`) - documents the key in `config.example.json:210` as a `_comment_clientId` entry rather than a value, because `docker/entrypoint.sh:6` copies that file as a live config and a literal value would give every default deployment the same ID. Also listed in `cmd/ingestor/README.md:94`. ## paho behaviour - No client-side length limit. `SetClientID` only stores the value; the 65535 check in `packets/connect.go:156` is in `Validate()`, which the client never calls. - The default ID is longer than the MQTT 3.1 limit of 23 characters for most source names. paho falls back to MQTT 3.1 after any refused CONNACK when no protocol version is set (`client.go:412`), so on a broker that refuses the first 3.1.1 attempt, the retry may hit that limit. I did not cap the length because paho does not require it and the 3.1.1 path accepts it (see below). ## Tests `cmd/ingestor/mqtt_opts_test.go:53-107`: - default ID is non-empty, has the sanitized name prefix, and contains only `[0-9A-Za-z-]` - broker host is used when the name is empty - configured `clientId` is used verbatim - two unconfigured sources with the same name get different IDs - the client built from the options reports the same ID Mutation checks: removing the random bytes fails the "different IDs" test; removing sanitization fails the prefix and character-set tests. `go test ./...` in `cmd/ingestor` passes except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails locally on Windows for a symlink privilege reason. `gofmt` and `go vet` are clean. ## Validation against a real broker On a staging instance (build `e84d2da6`) connecting to a Mosquitto bridge: ``` MQTT [lincomatic] connection attempt #1 to tcp://mosquitto-bridge:1883 MQTT [lincomatic] connected to tcp://mosquitto-bridge:1883 as client corescope-lincomatic-71a6eb MQTT [lincomatic] subscribed to meshcore/# ``` The 27-character default was accepted on the first attempt and packets kept arriving afterwards. ## Not verified - Only one broker type (Mosquitto) was tried. - `-race` was not run locally (no cgo toolchain on the test machine). - The case where both the source name and the broker host are empty (ID becomes `corescope-<hex>`) has no test. Fixes Kpa-clawbot#2013 ## Review follow-up (commit `a1d6709e`) An independent review found no bug in the ID handling, but the tests covered less than their names said. Changed, tests only (`cmd/ingestor/mqtt_opts_test.go`): - `TestBuildMQTTOpts_ClientIDSurvivesReconnects` replaces the old stability test, which only checked that paho copies the options. Against a loopback fake broker built on paho's `packets` codec, the first CONNECT, paho's auto-reconnect after the broker drops the socket, and the watchdog force-reconnect (`buildForceReconnectFn`) must all carry the same non-empty ID. It runs in about 0.01 s and passed `-count=30 -cpu 1,2,8`. - `TestBuildMQTTOpts_ClientIDDefaultShape` asserts full IDs: `^corescope-local-feed-1-[0-9a-f]{6}$`, `^corescope-mqtt-example-com-[0-9a-f]{6}$` for the broker host fallback (no port), and `^corescope-[0-9a-f]{6}$` with neither a name nor a host. - Mutations now caught: `SetClientID` removed, `u.Host` instead of `u.Hostname()`, a 1-byte suffix, the name guard dropped, sanitization removed. The "as client" log line has no test because it is logged from a closure inside `main()`. Corrections to the description: - **Fallback to MQTT 3.1.** paho falls back after any failed handshake once the socket is open, not only after a refused CONNACK: also a read error or timeout before any CONNACK, or a first packet that is not a CONNACK (`client.go:401-416`, `net.go:83-97`). A failed dial does not trigger it (`client.go:387-391`), and after the first successful connect the protocol version is locked in (`client.go:422-424`). Without this PR the 3.1 retry sent an empty ID, which MQTT 3.1 forbids as well, so nothing gets worse. - **Broker side.** On the EMQX broker we run, authorization has per-username and all-client rules and no client-ID rules (checked through its REST API). Two per-username rules use `${clientid}` in a topic, but both are publish rules and the ingestor only subscribes, so no rule can match it. A broker that caps IDs at 23 characters but accepted the empty ID before would now reject the default ID for source names of 7 or more characters; I have no evidence such a broker is in use. - The "Not verified" item about the empty name and host case no longer applies. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Refs Kpa-clawbot#2146 (first item: "Stop issuing SQL while holding `store.mu`"). ## Situation Several store functions ran SQLite queries while holding `PacketStore.mu`. The server's read pool has 4 connections (`db.go:153`), so a store-lock holder could wait for a pool connection. `mu` is a `sync.RWMutex`: once the ingest poller asks for the write lock, every new reader queues behind it, including `/api/healthz`. A test that scans the package (below) finds 16 such call sites on `master` at 9dbc287. ## My changes in this PR One commit per part: 1. **Caches serve stale while one refresh runs** (`4d9a818e`). `getCachedNodesAndPM`, `resolveRegionObservers` and `resolveAreaNodes` ran their SQL inline on a 30 s TTL miss, also from ingest under the write lock. Past the TTL they now return the cached value and start one background refresh per key (`cache_refresh.go`). - A cache that was never built, or was invalidated with `InvalidateNodeCache`, still loads inline. A generation counter stops a refresh that started before an invalidation from marking the cache fresh. - The region cache no longer holds `regionObsMu` across its query. Its refresh re-queries only the regions looked up since the previous refresh. That keeps it bounded, because the region comes from the query string. 2. **resolved_path after the lock** (`750dd951`). `QueryPackets`, `QueryMultiNodePackets`, `GetTransmissionByID`, `GetPacketByHash`, `GetPacketByID`, `GetObservationsForHash` and `GetNodeHealth` record the lookups under the lock (`rpJob`, with a snapshot of observation IDs and path JSON) and run them after releasing it. The number of queries per row is unchanged. 3. **Single queries before the lock** (`bf85e63e`): the `node=` filter (`resolveQueryNode`), multi-node pubkey resolution, the can_relay sets in node health, the observers read for hash-size home regions (`observerIATAs`), and the stats read in topology. 4. **Bulk health** (`93b7e8ec`): the nodes-table read runs before the lock. The region filter, which needs the in-memory indexes, is applied to those rows under the lock. 5. **`/paths` observations race** (`c30e9b43`, `fbf0d7ef`). `handleNodePaths` read each candidate's canonical resolved_path after releasing the lock, and that lookup walked `tx.Observations`, which ingest appends to under the write lock. `-race` reports it in the new test. The handler now copies the survivors' observation IDs and path JSON in a short read lock, and parses after releasing it. `fetchResolvedPathForTxBest` had no caller left and is replaced by `bestResolvedPath`. 6. **Topology node count** (`4094d215`). Topology used `db.GetStats()` for one field, the 7-day active node count. `GetStats` also runs `COUNT(*)` over transmissions and observations. That count is now `CountActiveNodes`, one query on `nodes`, and `GetStats` uses it too. ## Tests - `TestNoSQLUnderStoreLock` parses the package and fails on any call inside a `store.mu` section that reaches SQL, directly or through other package functions. It allows the three caches above as callees (they only query when cold) and the startup loaders as holders. `TestNoSQLUnderStoreLockDetects` runs the scan on a small synthetic source, so a scan that stops matching cannot pass silently. On master: 16 call sites. On this branch: 0. - `TestSlowResolvedPathSQLDoesNotStallReaders` reproduces the convoy. A test hook makes the resolved_path query take 300 ms; `/api/packets` sits in it, the ingest writer queues, then a reader takes `RLock`. With the lookup back under the lock (as on master) the reader waited 590 ms. On this branch it did not wait measurably. - `TestResolvedPathSQLRunsWithoutStoreLock` checks all seven functions from part 2. `cache_swr_test.go` covers stale serving, one refresh per key, inline rebuild after invalidation, and a cold region not blocking a cached one. `TestNodePathsDoesNotReadObservationsUnlocked` fails under `-race` with the old read. `TestCountActiveNodes` pins the 7-day count against `GetStats`. ## Verification - `cmd/server`: `go test ./...` green, and `go test -race` on the full package green with 0 races, both on the final commit. - E2E against a local server on the CI fixture: `test-e2e-playwright.js` 148/151 (3 skipped by the suite itself), plus `test-issue-1486-collapse-reopens-detail-e2e.js`, `test-issue-1692-packets-init-parallel-e2e.js` and `test-channel-issue-1087-e2e.js`, all passing. The other E2E suites were not run locally. - Browser: packets, node detail, analytics hash sizes and topology render. In the API responses, resolved_path is present on 44 of 50 packets, on packet detail, and on health `recentPackets`. No console errors. ### Staging measurement A staging instance with 129k transmissions in memory. Load ran on the same host: 4 threads looping `/paths`, `/api/packets?limit=200&expand=observations`, node health, `hash-sizes?region=`, `topology?region=` and `/api/packets?node=`, while one thread sampled `/api/healthz` every 200 ms. Each build had one cycle: 360 s during the ingestor's startup `ANALYZE` (heavy disk reads for 7 to 9 minutes after every restart), then 2 runs of 120 s after it finished. Values in ms. | | without this PR | with this PR | |---|---|---| | **During ANALYZE** | | | | `healthz` p95 / p99 / max | 156 / 1105 / 5681 | 42 / 905 / 4280 | | `topology?region` p99 / max | 2537 / 3061 | 1942 / 4159 | | `hash-sizes?region` max | 5809 | 2866 | | node health max | 5719 | 1110 | | `packets?expand` max | 3912 | 2570 | | calls over 5 s, excluding `/paths` | 2 | 0 | | **After ANALYZE** (two runs) | | | | `healthz` p95 / max | 444-667 / 2139-4356 | 189-335 / 1377-3238 | | `topology?region` max | 3401-5363 | 1440-1927 | | `/paths` p50 | 1466-4736 | 1125-2648 | | calls over 5 s | 3 | 0 | The two builds were measured about 50 minutes apart, so the background traffic was not identical. An earlier version of this branch, without `fbf0d7ef`, raised `healthz` p95 to 2.7-2.9 s in the same setup, because it parsed every `/paths` candidate's path JSON under the read lock. Without `4094d215`, `topology?region` reached 8-17 s during `ANALYZE` in three cycles. The table is for the final branch. ## Not in this PR - The query count of `/paths` (1 to 3 per candidate, `resolved_index.go:160`). It still dominates that endpoint, at p95 8 to 15 s on staging. It is its own item in Kpa-clawbot#2146. - Batching the resolved_path queries. - The other per-endpoint items in Kpa-clawbot#2146 (reach, channels, perf, observers, config/regions, battery, neighbors). - A cache's first lookup still queries inline, under the lock where the caller holds it. That happens once per key per process, by design, and the invariant test allows it explicitly. - `cmd/ingestor` is unchanged. No measurement on an instance the size of the ones in Kpa-clawbot#2146. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…pa-clawbot#2155) Refs Kpa-clawbot#2146 (item: `/api/nodes/:pubkey/reach`). ## Situation The reach scan joins `observations` to `transmissions` and filters on `o.timestamp >= ?` plus one `o.path_json LIKE '%"XX"%'` per token. With the planner statistics `ANALYZE` produces on a large database, SQLite runs it as `SCAN t`: every transmission, with an index probe into `observations` (`idx_observations_tx_ts`) per row. The cost is then the size of `transmissions`, whatever node is asked for and whatever the window. ## My change in this PR `scanReachRows` uses `CROSS JOIN` for the transmissions join. In SQLite that fixes the join order: `observations` is the outer loop, read through `idx_observations_timestamp` for the window, and each row looks up its transmission by primary key. Results are the same rows; it is still an inner join. The query text moves into `reachScanSQL(nTokens)`, so a test can inspect the plan. ## Measurements Read-only on a database with 18.3M observations and 1.5M transmissions (Python `sqlite3` 3.45.1 on the host), 7-day window: | node (rows returned) | `JOIN`, first run | `JOIN`, warm | `CROSS JOIN`, both runs | |---|---|---|---| | busiest relay (54,387) | 10.29 s | 3.27 s | 1.95 s | | quartile (3,913) | 3.16 s | 3.15 s | 1.81 s | | median (1,310) | 3.17 s | 3.18 s | 1.82 s | | a repeater reported as timing out in the browser (34,209) | 14.11 s | 3.25 s | 1.93 s | On that instance `/api/nodes/:pubkey/reach` reports p95 188.6 s and max 192.6 s over 34 calls in `/api/perf`; a single uncached call from the host took 3.4 to 3.6 s, so most of that is waiting for the 2 build slots and the read pool. On a staging instance (smaller database), with unique cache keys and `days` 6 or 7, one cycle per build after the ingestor's startup `ANALYZE` had finished: | | without this PR | with this PR | |---|---|---| | sequential, 12 calls, p50 | 3402-3801 ms | 2078-2467 ms | | 2 concurrent clients (= `reachMaxConcurrentBuilds`), p50 / p95 | 3575-4087 / 4049-5291 ms | 2271-2462 / 2814-2928 ms | | calls completed in 120 s | 58-68 | 96-101 | The staging builds also contain Kpa-clawbot#2147. ## Verification - `TestReachScanStartsFromObservationsTimeIndex` creates the two production indexes, loads the `sqlite_stat1` values from the 18.3M-observation database, and checks that the plan starts with `SEARCH o USING INDEX idx_observations_timestamp`. With the old `JOIN` it fails with the production plan (`SCAN t`, then `SEARCH o USING INDEX idx_observations_tx_ts`). - The existing reach tests (`TestScanReachRows_DecodesRows`, `TestScanReachRows_CapTruncates`, attribution tests) pass unchanged. - `cmd/server`: `go test ./...` green. ## Not in this PR - The scan still reads every observation in the window and applies the `LIKE` per row, so its cost grows with the window and with traffic. Removing the scan needs an index on hop tokens (for example a table the ingestor maintains). I have not measured what that would cost in storage. - The build-slot limit (2) and the 5-minute cache are unchanged. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
n30nex
marked this pull request as ready for review
October 10, 2026 04:35
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…odex/canadaverse-ui
…ktrees/postgres-migration/Corescope into codex/canadaverse-ui
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Apply Canadaverse's identity throughout this CoreScope fork: shared emblem and wordmark, home screen, navigation, controls, panels, and accessible dark/light palettes. Operator branding overrides, theme resets, packet indicators, and mobile navigation remain supported. Narrow-screen replay and navigation controls retain their touch targets with fallback fonts.
This PR is based on
codex/postgres-migrationat01b2a941feb9210b37a72eeab4d196b106bbadb4, so its diff contains only branding and the associated regression coverage. PostgreSQL, account management, and CoreDrive RX behavior are retained.Validation:
504f44d: SQLite/PostgreSQL backend jobs, six browser groups, both race checks, and two-architecture image builds with packaged PostgreSQL startup checks.Performance: shared CSS and one cached SVG asset; no added polling, per-packet calls, data transformations, or continuous decoration. Packet, map and ingestion data paths are unchanged.
Deployed and verified at https://dev.canadaverse.org on exact commit
504f44d1373f668245c7b2953748ae34f586ab59. The public home, 24-hour packet view and performance dashboard were checked in a real browser with no console errors. The PostgreSQL startup-memory and history-query repairs come from the base branch. The application peaked at 592.47 MiB against its 1,536 MiB limit during verification, with no OOM events or restarts. Both Canada feeds are fresh, CoreDrive RX is connected, and database write errors are zero. Existing accounts were preserved and other containers retained their identities and start times during cutover. Private configuration and native PostgreSQL backups are retained for rollback. Large historical packet requests still take about 18–22 seconds; overnight stability has not yet been observed on this revision.Integration target: the fork's
master, including the PostgreSQL foundation already validated in the deployed development stack. The upstream PostgreSQL contribution remains on its existing branch.