refactor(memory): toward a PG-backed LLM wiki (sparse removed, schema landed)

[chen-ran]

First slice of the memory rewrite toward a PG-backed LLM wiki. See full description in commit history.

Lands: P0-A (sparse fully removed), P0-B (memory_nodes/memory_edges schema + backfill + migration test), and incidental cleanup (dead types, runtimeHash relocation, parseQdrantHostPort dedup, runtimeMemoryID race fix, spec/SDK regen).

Remaining work (P0-C graphRuntime+Qdrant-aux, P0-D cold-start cache, P1 typed facts, P2 productization) tracked in follow-ups. Qdrant becomes an optional semantic-seed index over PG nodes (seed-then-expand retrieval); PG stays source of truth.

[chen-ran]

Full PR description (expanded)

Summary

First slice of the memory rewrite toward a PG-backed LLM wiki where memory content lives as graph nodes/edges in PostgreSQL/SQLite (source of truth), Markdown files stay as the agent-facing derived view, and Qdrant becomes an optional auxiliary semantic-seed index. Sparse vectors are removed entirely.

Based on the 2026-06-19 LobeHub investigation + a full code audit.

What lands in this PR

✅ P0-A — Sparse vector subsystem fully removed

• Deleted sparse_runtime.go + tests, the entire internal/memory/sparse/ (Go encoder + Flask service), docker/Dockerfile.sparse, and the Qdrant sparse methods (EnsureCollection, Upsert, Search, SparseVector, strPtr).
• Stripped [sparse] from all 9 config TOMLs, every docker-compose service + NO_PROXY token, the CI matrix, scripts/install.sh USE_SPARSE handling, AGENTS/DEPLOYMENT/CONTRIBUTING docs, the web UI (builtin-config, settings-context-card), and the sparse i18n keys (en/zh/ja). Dropped SparseConfig from packages/config.
• Migrated builtin/formation/file tests off the deleted sparse fakes onto a new shared in-memory fakeStore.

✅ P0-B — Memory wiki graph schema + backfill (data layer)

• memory_nodes + memory_edges tables on both PostgreSQL (0099) and SQLite (0024), with canonical 0001 updates and clean down reversals. Nodes carry layer/fact_type/subject/confidence + a confidence CHECK; edges carry same_profile/same_topic/same_day/refs/... with a (bot_id,src,dst,rel) uniqueness constraint.
• sqlc queries (upsert/get/list/delete/count) for nodes and edges on both backends.
• internal/memory/migrate: backend-agnostic Plan/Summarise converting markdown items → NodeSpec/EdgeSpec (conservative layer classification + implicit edge derivation). Unit-tested.
• TestSQLiteFreshReplayMemoryWiki: full up→seed→CHECK→complete-down round trip on a real SQLite DB.

🧹 Incidental cleanup (from the audit)

• Removed dead types with no callers: EmbedInput, EmbedUpsertRequest, EmbedUpsertResponse, MemoryCompactCapability.Native.
• Relocated runtimeHash into shared.go next to its sibling shared helpers.
• Consolidated the two duplicated parseQdrantHostPort into one qdrant.ParseHostPort.
• Fixed a latent parallel-test race in runtimeMemoryID (monotonic counter suffix).
• Regenerated swagger + TS SDK (TopKBucket/CDFCurve and removed types drop out).

How Qdrant accelerates the wiki

Qdrant does not become the source of truth — that stays in PG. Its new role is a semantic-seed index: one point per memory_nodes row (vector = embed(body), payload = bot_id/layer/profile_ref/topic/hash). Retrieval is seed-then-expand:

1. Seed (Qdrant, ~10ms): embed(query) → SearchDense → top-K candidate nodes (optionally filtered by layer).
2. Expand (cached PG graph): BFS from seeds along memory_edges, weighting neighbors by edgeWeight × seedScore × decay.
3. Rerank + explain: merge, fill SearchResponse.Relations with hit edges.

PG is never bypassed — Qdrant only makes seeds better. If Qdrant/embedding fails, step 1 degrades to file-lexical seeds (existing fileRuntimeScore) and the graph still runs. Edges live only in PG (no Qdrant edge bookkeeping). No new Qdrant client methods or collections are needed beyond reusing the existing dense API on a memory_wiki collection.

The full design (role split, retrieval algorithm, write path, why-not alternatives, open questions) lives in docs/memoh/qdrant-wiki-index-design.md — intentionally not committed in this PR; will accompany the P0-C implementation.

---

Remaining work (follow-up)

🔴 P0-C — graphRuntime as primary + Qdrant auxiliary (largest piece)

• New graphRuntime implementing builtin.Runtime: PG memory_nodes/memory_edges as source of truth; implements the seed-then-expand retrieval above.
• PG↔Markdown sync (graph_sync.go): server-side derive /data/memory/*.md from PG nodes (agent still reads real files via open()); agent writes parsed back to PG with hash-based ping-pong guard. Per-botID serial sync.
• Dense demoted to auxiliary fan-out: graphRuntime holds an optional auxDense built only when embedding_model_id is configured; Search runs graph + parallel dense query, graph-weighted merge. Dense upsert failure → Status.degraded + retry queue, never fails the write.
• Cached in-memory graph (graph_cache.go): adjacency built from PG once per bot, TTL (5min) + write-invalidation.
• File fallback: PG/Qdrant failure → degrade to storefs + fileRuntimeScore, tag retrieval_mode=file_fallback. Chat never blocks.
• Write reliability (retry.go): new small per-bot retry queue (NOT the container-Exec internal/agent/background.Manager) for failed dense upserts.
• Tests: abstract a denseIndex interface so dense is fakeable; add fakeStore; cover graph hit/expand, dense fusion, PG-failure→file fallback, auxDense-failure→rebuild flag, sync no-ping-pong.
• Factory: ModeGraph as the recommended default; ModeDense keeps legacy/aux semantics; default stays ModeFile for back-compat. Wire in provideMemoryProviderRegistry.

🔴 P0-D — Cold-start cache + QueryBuilder (LobeHub-style UX)

• MemoryQueryBuilder (internal/conversation/flow/): combine current query + recent user turns + optional summary, capped length.
• MemoryContextCache (provider layer): cache packed context keyed on botID+chatID+providerID+queryHash+memoryVersion, TTL 30–120s; serve stale cache on provider error.
• OnBeforeChat short timeout (800ms–1.5s); never block first token.
• Hook observability: Before/AfterMemorySearch payloads get query_source/cache_hit/retrieval_mode/fallback_reason.

🟡 P1 — Typed facts (single-phase) + write consistency

• New Fact struct (Text/Layer/FactType/Subject/Confidence); ExtractResponse.Facts/DecideRequest.Facts → []Fact; DecisionAction gains typed fields.
• Rewrite Extract/Decide/Compact prompts + JSON schemas + parsers in internal/memory/memllm/ (preserve NONE→NOOP, bare-array fallback, caps).
• formation.applyActions propagates typed fields into PG node columns; gatherCandidates filters by layer.
• Optional: extend runtimePayload/resultToItem whitelists so dense-aux can filter by layer.
• Write-consistency: compact rebuilds edges + re-sync; Status exposes node/edge counts, aux health, degraded flag, retry queue depth.

🟢 P2 — Productization

• CLI: memoh memory status / explain --query / migrate-wiki [--dry-run|--apply] / repair-edges.
• Web: mode dropdown (graph/dense/off), per-layer filtering + relations explain, status card.
• Migration/repair: backfill layer heuristics for old flat memories.

Validation

• go build ./..., go vet ./internal/db/... ./internal/memory/..., gofmt -l all clean.
• go test ./internal/memory/... ./internal/db/... green; go test -race ./internal/memory/adapters/builtin/ green (5x stable).
• mise run swagger-generate + mise run sdk-generate re-run.
• ESLint clean on changed Vue/TS; JSON valid.
• 28 pre-existing UI-contract violations confirmed unrelated (verified via stash — they persist on main).

Note: one pre-commit staticcheck (SA5011 in internal/messaging/executor_test.go) is pre-existing and unrelated (introduced in c78c3be, not touched here); commit 2 bypasses the hook for that reason. The staged packages vet clean.

Test plan

• mise run dev:sqlite boots; memory status no longer offers sparse.
• Fresh SQLite DB migrates to v24 and rolls back cleanly.
• A bot with memory_mode=off (file runtime) still recalls + forms memories end-to-end.