From a95e0eac880fcb62f9f6be459a68fc2ac48cbf2a Mon Sep 17 00:00:00 2001 From: Charlie Coutts Date: Sun, 5 Jul 2026 00:21:57 +0000 Subject: [PATCH] Make the debugging skill aware of doltlite backends (agnostic, no editorializing) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pack modeled the data plane as the Dolt mysql sql-server (PROCESSLIST, a TCP port, DOLT_GC over the wire) and treated doltlite as a raw SQLite file. It had no notion of the DoltLite backend-plugin deployment: bd/gc run plain (unlinked) and speak to `bd-backend-doltlite serve` (stdio) + `gc-doltlite-fastpath serve`, store at `.beads/doltlite/*.db`, no TCP server. Keep the skill backend-agnostic — its job is to detect a store's backend and trace to the OWNING LAYER's repo, not to rank backends: - gas-stack-map: a "detect the backend FIRST" recipe (`.beads/metadata.json` is ground truth) listing the common backends + their repos, explicitly leaving the set open (others may exist). No official/default/legacy framing. - gc-diagnostic-toolkit: tag the sql-server section as dolt-only; give the doltlite section the plugin model — query via `bd sql` (not a raw `sqlite3` open), inspect the `…serve` procs + store dir, use DoltLite SQL maintenance functions; defer to the `beads-doltlite` skill for dolt_gc/flatten/locks/fastpath. - SKILL: reflect the pluggable backend in the stack tree. Co-Authored-By: Claude Opus 4.8 --- skills/debugging-gas-issues/SKILL.md | 2 +- .../references/gas-stack-map.md | 19 +++++++++++++++---- .../references/gc-diagnostic-toolkit.md | 19 +++++++++++++------ 3 files changed, 29 insertions(+), 11 deletions(-) diff --git a/skills/debugging-gas-issues/SKILL.md b/skills/debugging-gas-issues/SKILL.md index 56788ed..b7b5443 100644 --- a/skills/debugging-gas-issues/SKILL.md +++ b/skills/debugging-gas-issues/SKILL.md @@ -33,7 +33,7 @@ gc gastownhall/gascity · gastownhall/gastown ├─ dolt dolthub/dolt (mysql-protocol, sql-server, port-based) │ └─ gms dolthub/go-mysql-server (the SQL engine — planning, indexes) │ └─ vitess dolthub/vitess (MySQL protocol/parser) - ├─ doltlite dolthub/doltlite (SQLite-backed — different diagnostics, no sql-server) + ├─ doltlite dolthub/doltlite (SQLite-backed; linked OR backend-plugin = bd-backend-doltlite + gc-doltlite-fastpath — no sql-server/port) └─ driver dolthub/driver ``` diff --git a/skills/debugging-gas-issues/references/gas-stack-map.md b/skills/debugging-gas-issues/references/gas-stack-map.md index 0084929..b7cc84f 100644 --- a/skills/debugging-gas-issues/references/gas-stack-map.md +++ b/skills/debugging-gas-issues/references/gas-stack-map.md @@ -13,7 +13,7 @@ The symptom surfaces high in the stack (a `gc` command feels slow, an agent stal | go-mysql-server (gms) | `dolthub/go-mysql-server` | The SQL engine embedded in dolt: query planning, index selection, joins, expression eval | a dolt dependency (check dolt's `go.mod`) | | vitess | `dolthub/vitess` | MySQL wire protocol + SQL parser used by gms | a gms dependency | | driver | `dolthub/driver` | The Go `database/sql` driver gc/bd use to talk to a dolt server | a gc/bd dependency | -| doltlite | `dolthub/doltlite` (`doltlite-python`) | SQLite-backed version-controlled store — an alternative to dolt; **no mysql sql-server, no port** | the consumer's build cache; an on-disk SQLite file | +| doltlite | `dolthub/doltlite` + the beads **backend plugin** (`bd-backend-doltlite`) & gc **fastpath** (`gc-doltlite-fastpath`) | SQLite-backed version-controlled store — an alternative to dolt; **no mysql sql-server, no port**. In the *plugin* deployment bd/gc are plain (unlinked) and talk to a `…serve` subprocess over stdio | `.beads/doltlite/*.db`; the co-located plugin binaries (e.g. `~/.local/lib/beads-plugin/`); `.beads/metadata.json` names them | ## Ground-truth sources (consult before tracing or changing behavior) @@ -43,9 +43,20 @@ git -C log -1 --format='%H %ci %s' # source HEAD # mismatch -> your trace may not reflect the binary; rebuild or check out the right commit ``` -## dolt vs doltlite (don't assume mysql) +## Which data plane? — detect the backend FIRST (don't assume mysql) -- **dolt**: a MySQL-protocol `sql-server` on a TCP port. Diagnose with `SHOW PROCESSLIST`, `information_schema.processlist`, global status, `ss`/`lsof` on the port. Port resolution: `--port` flag > city `dolt.port` config > `/.beads/dolt-server.port` file > legacy default. -- **doltlite**: a version-controlled **SQLite** file. **No server, no port, no PROCESSLIST.** Diagnose with SQLite tooling against the file (`.dolt`/`.doltlite` dir), file size on disk, and the consuming process's own profiling. The CPU-vs-load, binary-grep, bead-store-layout, and dogfood techniques still apply; the *server* techniques do not. Check which backend the city/store actually uses before reaching for `SHOW PROCESSLIST`. +A beads store can sit on any of several data-plane backends, and their diagnostics differ completely — so identify the store's backend and **trace to that layer's repo** (the stack table above). `.beads/metadata.json` is the ground truth — read it *before* reaching for `SHOW PROCESSLIST`. The common ones are below; the list isn't closed (others may exist), so key off metadata, not a fixed set: + +```bash +jq -r '.backend, (.backend_plugin_command // "—")' .beads/metadata.json # backend + plugin cmd (or —) +ls -d .beads/dolt .beads/doltlite 2>/dev/null # which store dir exists +pgrep -af 'dolt sql-server|bd-backend-doltlite|gc-doltlite-fastpath' # which serve procs are live +``` + +- **dolt (sql-server)** — `backend: dolt`; a MySQL-protocol server on a TCP port, store under `.beads/dolt`. Diagnose with `SHOW PROCESSLIST`, `information_schema.processlist`, global status, `ss`/`lsof` on the port. Port resolution: `--port` > city `dolt.port` > `/.beads/dolt-server.port` > legacy default. +- **doltlite, linked** — `backend: doltlite`, **no** `backend_plugin_command`; DoltLite compiled *into* bd/gc. Version-controlled, SQLite-backed; **no server, no port, no PROCESSLIST.** +- **doltlite, backend-plugin** — `backend: doltlite` **with** `backend_plugin_command` set. bd/gc are plain (unlinked) and launch `bd-backend-doltlite serve` (bd storage, over stdio, per-invocation) + `gc-doltlite-fastpath serve` (gc's always-on read fastpath). Store is `.beads/doltlite/*.db`; the plugin binaries are co-located (metadata names them). **No TCP server** — you cannot `SHOW PROCESSLIST` or `ss` a port. Inspect via `bd sql` (routed through the plugin), the `…serve` processes, the store dir on disk, and the DoltLite SQL maintenance functions. Deep commands + gotchas (`dolt_gc`, flatten, maintenance, locks, native read fastpath) live in the DoltLite contract itself (`dolthub/doltlite` + its README) and the plugin repos (`duncan4123/beads-backend-doltlite`, `duncan4123/gascity`). A city that imports the `beads-doltlite` pack (`gastownhall/gascity` `//examples/beads-doltlite`) also gets its `doltlite` skill as a convenience layer over those. + +Detect which backend a given store uses and **trace to that owning layer (and its repo)** — don't assume. On a shared box the backends can coexist **per city** (one city on one backend, a throwaway on another), so detect per store, never assume per host. See `gc-diagnostic-toolkit.md` for the concrete commands per backend. diff --git a/skills/debugging-gas-issues/references/gc-diagnostic-toolkit.md b/skills/debugging-gas-issues/references/gc-diagnostic-toolkit.md index 3698906..52de244 100644 --- a/skills/debugging-gas-issues/references/gc-diagnostic-toolkit.md +++ b/skills/debugging-gas-issues/references/gc-diagnostic-toolkit.md @@ -40,7 +40,9 @@ pidstat 3 1 | awk '/^Average:/ && $3 ~ /^[0-9]+$/ {cpu[$NF]+=$8} END{for(c in cp ``` Aggregates sustained CPU by command. (`ps` %CPU is lifetime-average — misleading for "now".) -## dolt server introspection +## dolt server introspection (backend: dolt only — confirm first) + +Applies to the **dolt sql-server** backend. On a doltlite backend there is no server/port/PROCESSLIST — skip to the doltlite section. Confirm with `jq -r .backend .beads/metadata.json`. ```bash PORT=$(cat /.beads/dolt-server.port 2>/dev/null || echo 3307) # resolution: --port > city dolt.port > port-file > legacy @@ -68,11 +70,16 @@ gc dolt-cleanup --probe --json # orphan dbs / stale procs (NEVER --force wit ``` Disk reclaim after deleting rows is a **dolt** GC: `CALL DOLT_GC('--full')` (online-safe; quiesce writers first). `gc dolt compact` gates on commit count and skips low-commit/high-churn dbs. -## doltlite (SQLite backend) +## doltlite (embedded backend — linked or plugin) -No server/port/PROCESSLIST. Inspect the file: +No server/port/PROCESSLIST. First confirm which doltlite deployment (see `gas-stack-map.md` → "Which data plane?"): +```bash +jq -r '.backend, (.backend_plugin_command // "—")' .beads/metadata.json # doltlite + (plugin cmd | —) +pgrep -af 'bd-backend-doltlite|gc-doltlite-fastpath' # plugin serve procs (plugin model) +du -sh .beads/doltlite 2>/dev/null # store size on disk +``` +Query **through bd** (routed to the backend — works for linked and plugin), not a raw `sqlite3` open of the `.db` (that bypasses the working-set/commit model and can read a stale or locked view): ```bash -du -sh /.doltlite 2>/dev/null -sqlite3 'SELECT issue_type, COUNT(*) FROM issues GROUP BY issue_type;' # adapt to the actual schema/path +bd sql -q "SELECT issue_type, COUNT(*) n, SUM(status='closed') closed FROM issues GROUP BY issue_type ORDER BY n DESC" ``` -The CPU-vs-load, binary-grep, store-layout, and dogfood techniques still apply; the server techniques do not. **Check the backend before reaching for `SHOW PROCESSLIST`.** +Disk reclaim / maintenance is a **DoltLite** operation, not `CALL DOLT_GC` over a wire — use the DoltLite SQL functions (`SELECT dolt_gc();`, flatten, maintenance) and mind the store locks. The authoritative references for those, the native read fastpath, and DoltLite lock/maintenance gotchas are the DoltLite contract (`dolthub/doltlite` + README) and the plugin repos (`duncan4123/beads-backend-doltlite`, `duncan4123/gascity`); the `beads-doltlite` pack's `doltlite` skill (if your city imports it) collects them. The CPU-vs-load, binary-grep, store-layout, and dogfood techniques still apply; the server techniques (`PROCESSLIST`, `ss` on a port, `DOLT_GC('--full')` over the wire) do not. **Check the backend before reaching for `SHOW PROCESSLIST`.**