|
| 1 | +# Plan — Storage rebalance: drop FTS5, persist vectordb, cap log search |
| 2 | + |
| 3 | +**Status:** Approved scope, ready for implementation |
| 4 | +**Date:** 2026-04-30 |
| 5 | +**Reviewers:** codex (vectordb persistence design, 2026-04-30); user (scope sign-off, 2026-04-30) |
| 6 | + |
| 7 | +## Problem |
| 8 | + |
| 9 | +Three coupled inefficiencies surfaced while sizing for the 150-service × 7-day production target on SQLite: |
| 10 | + |
| 11 | +1. **FTS5 inverted index consumes 30-40% of DB disk.** At the current 4GB sample, ~1.5GB; at full production scale, tens of GB. The UI never queries it (verified: `ui/src/hooks/useLogs.ts:21` calls `/api/logs` with no `q`); only the `search_logs` MCP tool depends on it, and that has a working LIKE fallback already wired at `internal/storage/log_repo.go:105`. |
| 12 | +2. **vectordb is not persisted.** Daemon restart empties the in-memory TF-IDF index. `find_similar_logs` and GraphRAG `SimilarErrors` return degraded results until ERROR/WARN logs flow back in naturally — minutes to hours, depending on error rate. |
| 13 | +3. **`search_logs` has no time-range cap.** Tolerable with FTS5; without it, an unscoped 7-day keyword query is a 30+ minute table scan. Worst case must be bounded. |
| 14 | + |
| 15 | +## Goal |
| 16 | + |
| 17 | +Cut SQLite footprint by 30-40%, eliminate vectordb cold-start blindness, and bound `search_logs` worst case to single-digit seconds — without changing `find_similar_logs`, GraphRAG, or any non-search MCP tool surface. |
| 18 | + |
| 19 | +## Non-goals |
| 20 | + |
| 21 | +- Inverted index / sublinear cosine search (requires raising 100k cap, separate change) |
| 22 | +- Dense embeddings / ONNX |
| 23 | +- Per-tenant snapshot sharding, multi-process flock |
| 24 | +- Raising `VECTOR_INDEX_MAX_ENTRIES` |
| 25 | +- Removing `search_logs` (kept, degraded via LIKE) |
| 26 | +- Changes to Postgres `pg_trgm` path (SQLite-only) |
| 27 | + |
| 28 | +--- |
| 29 | + |
| 30 | +## Phase 0a — `LOG_FTS_ENABLED` config flag |
| 31 | + |
| 32 | +Default `false`. New SQLite deploys skip FTS5 entirely. |
| 33 | + |
| 34 | +**Files:** |
| 35 | +- `internal/config/config.go` — add `LogFTSEnabled bool`, env `LOG_FTS_ENABLED`, default false |
| 36 | +- `internal/storage/fts5.go` — `fts5Available()` returns false when flag is off; `EnsureLogsFTS5()` becomes no-op |
| 37 | +- `internal/storage/log_repo.go` — Repository gains `ftsEnabled bool`; `useFTS5` checks both driver + flag |
| 38 | +- `internal/storage/factory.go` (or wherever `NewRepository` is) — thread `cfg.LogFTSEnabled` through |
| 39 | +- `CLAUDE.md` — update FTS5 description, note opt-in |
| 40 | + |
| 41 | +LIKE fallback already exists at `log_repo.go:105-108`; flipping the flag silently routes `filter.Search` queries through it. No new fallback code. |
| 42 | + |
| 43 | +## Phase 0b — One-shot FTS5 reclaim |
| 44 | + |
| 45 | +For existing DBs with `logs_fts`: |
| 46 | + |
| 47 | +- New admin endpoint `POST /api/admin/drop_fts` (auth-gated, alongside existing `/api/admin/purge` + `/api/admin/vacuum`) |
| 48 | +- Body: `DROP TRIGGER IF EXISTS logs_fts_ai; logs_fts_ad; logs_fts_au; DROP TABLE IF EXISTS logs_fts; VACUUM;` |
| 49 | +- Returns `{"reclaimed_bytes": N, "elapsed_ms": M}` |
| 50 | +- Refuses (405) if `LOG_FTS_ENABLED=true` — won't drop while in active use |
| 51 | + |
| 52 | +**Files:** |
| 53 | +- `internal/api/admin_handlers.go` (or wherever `handlePurge`/`handleVacuum` live) — add `handleDropFTS` |
| 54 | +- `internal/api/server.go:108-109` — register route |
| 55 | +- Test: integration test creates `logs_fts`, calls handler, asserts gone + bytes reclaimed |
| 56 | + |
| 57 | +VACUUM blocks writes 10-60min on a 4GB DB — opt-in only, document for off-hours. |
| 58 | + |
| 59 | +## Phase 0c — 24h cap on `search_logs` |
| 60 | + |
| 61 | +**Description update at `internal/mcp/tools.go` ~line 37:** |
| 62 | + |
| 63 | +> "Search log bodies by keyword. Limited to the last 24 hours. Strongly recommend setting `service_name` and/or `severity` to scope the search; unscoped keyword queries scan large row counts without FTS5. Returns up to `limit` results ordered by timestamp desc." |
| 64 | +
|
| 65 | +**Helper at `internal/mcp/tools.go` (or new `internal/mcp/clamp.go`):** |
| 66 | + |
| 67 | +```go |
| 68 | +func clampTo24h(start, end, now time.Time) (time.Time, time.Time, error) { |
| 69 | + if end.IsZero() { end = now } |
| 70 | + if start.IsZero() { start = end.Add(-24 * time.Hour) } |
| 71 | + if end.After(now) { end = now } |
| 72 | + cutoff := now.Add(-24 * time.Hour) |
| 73 | + if end.Before(cutoff) { return time.Time{}, time.Time{}, errors.New("search window must be within the last 24h") } |
| 74 | + if start.Before(cutoff) { start = cutoff } |
| 75 | + if !start.Before(end) { return time.Time{}, time.Time{}, errors.New("start_time must be before end_time") } |
| 76 | + return start, end, nil |
| 77 | +} |
| 78 | +``` |
| 79 | + |
| 80 | +**Apply at:** |
| 81 | +- `internal/mcp/tools.go::toolSearchLogs` (~line 445) — after arg parse |
| 82 | +- `internal/api/log_handlers.go::handleGetLogs` — when `q` param non-empty (defense in depth; UI doesn't hit this with `q` but direct HTTP callers shouldn't bypass) |
| 83 | + |
| 84 | +**Tests:** 4 cases for `clampTo24h` (defaults, oversize window clamped, out-of-cap rejected, invalid range rejected) + 1 integration test each for MCP and HTTP. |
| 85 | + |
| 86 | +**Behavior matrix:** |
| 87 | + |
| 88 | +| Input | Effective window | |
| 89 | +|---|---| |
| 90 | +| nothing | now-24h → now | |
| 91 | +| start = 5d ago | now-24h → now (start clamped) | |
| 92 | +| start = 5d ago, end = 4d ago | rejected | |
| 93 | +| end = future | start → now (end clamped) | |
| 94 | +| `q=""` (no search) | unconstrained — cap only fires when search is set | |
| 95 | + |
| 96 | +--- |
| 97 | + |
| 98 | +## Phase 1-5 — vectordb persistence |
| 99 | + |
| 100 | +### Format |
| 101 | + |
| 102 | +``` |
| 103 | +bytes[0:4] magic "VDB1" (ASCII) |
| 104 | +bytes[4:8] format version uint32 BE |
| 105 | +bytes[8:12] payload CRC32-IEEE uint32 BE (over bytes[12:]) |
| 106 | +bytes[12:] gob payload Snapshot |
| 107 | +``` |
| 108 | + |
| 109 | +```go |
| 110 | +type Snapshot struct { |
| 111 | + LastIndexedID uint // max Log.ID seen by Add() |
| 112 | + MaxSize int |
| 113 | + Docs []LogVector |
| 114 | + IDF map[string]float64 |
| 115 | + WrittenAt int64 // unix seconds |
| 116 | +} |
| 117 | +``` |
| 118 | + |
| 119 | +### Tail-replay key |
| 120 | + |
| 121 | +**Use `Log.ID` (auto-increment PK), not `Log.Timestamp`.** Codex caught: `Index.Add()` has no dedup; timestamp-based replay would double-index boundary entries on every restart. ID is monotonic, dedup-free, DB-agnostic. |
| 122 | + |
| 123 | +Query: `WHERE id > ? AND severity IN ('ERROR','WARN','WARNING','FATAL','CRITICAL') ORDER BY id ASC LIMIT 10000`. Paged loop until empty. |
| 124 | + |
| 125 | +### Atomic write |
| 126 | + |
| 127 | +`writeAtomic(path, data)`: write to `path+".tmp"` → fsync → close → rename. On `EXDEV` (cross-device): log warn once, fall back to `os.WriteFile` directly. On fsync error: delete `.tmp`, return error. |
| 128 | + |
| 129 | +### Lifecycle |
| 130 | + |
| 131 | +**Startup** (in `cmd/main.go` after DB open, before ingest accept): |
| 132 | +```go |
| 133 | +idx := vectordb.New(cfg.VectorIndexMaxEntries) |
| 134 | +if cfg.VectorIndexSnapshotPath != "" { |
| 135 | + _ = idx.LoadSnapshot(cfg.VectorIndexSnapshotPath) // tolerates missing/corrupt |
| 136 | + _ = idx.ReplayFromDB(ctx, repo) // catches tail since LastIndexedID |
| 137 | + go idx.SnapshotLoop(ctx, path, cfg.VectorIndexSnapshotInterval) |
| 138 | +} |
| 139 | +``` |
| 140 | + |
| 141 | +**Shutdown:** final `idx.WriteSnapshot()` between gRPC/HTTP `Shutdown()` returning and `graphrag.Stop()` — captures every `Add()` that completed before ingest stopped. |
| 142 | + |
| 143 | +### Phases |
| 144 | + |
| 145 | +- **Phase 1**: `internal/vectordb/snapshot.go` + tests — encode/decode, magic/version/CRC, atomic write, EXDEV fallback. Pure isolation, no behavior change. |
| 146 | +- **Phase 2**: wire into `Index` — add `lastIndexedID` field tracked in `Add()`; methods `LoadSnapshot`, `WriteSnapshot`, `LastIndexedID()`. Config fields added but no goroutine yet. |
| 147 | +- **Phase 3**: DB tail replay — `Repository.LogsForVectorReplay(ctx, sinceID, limit) ([]Log, error)` (paged); `Index.ReplayFromDB(ctx, repo)` walks pages calling `Add()`. |
| 148 | +- **Phase 4**: `Index.SnapshotLoop(ctx, path, interval)` background goroutine; wire startup/shutdown in `cmd/main.go`. |
| 149 | +- **Phase 5**: 5 Prometheus metrics (`snapshot_writes_total{result}`, `snapshot_duration_seconds`, `snapshot_size_bytes`, `snapshot_load_total{result}`, `replay_logs_total`); CLAUDE.md vectordb section + config table. |
| 150 | + |
| 151 | +--- |
| 152 | + |
| 153 | +## Files (combined) |
| 154 | + |
| 155 | +### New |
| 156 | +- `internal/vectordb/snapshot.go` |
| 157 | +- `internal/vectordb/snapshot_test.go` |
| 158 | +- `internal/mcp/clamp.go` + `clamp_test.go` |
| 159 | + |
| 160 | +### Modified |
| 161 | +- `internal/vectordb/index.go` — `lastIndexedID`, `LoadSnapshot`, `WriteSnapshot`, `SnapshotLoop`, `ReplayFromDB` |
| 162 | +- `internal/storage/fts5.go` — gate on flag |
| 163 | +- `internal/storage/log_repo.go` — Repository ftsEnabled field |
| 164 | +- `internal/storage/repository.go` — `LogsForVectorReplay` |
| 165 | +- `internal/storage/factory.go` — config wiring |
| 166 | +- `internal/api/log_handlers.go` — 24h cap symmetry on `q=` |
| 167 | +- `internal/api/admin_handlers.go` (or equivalent) — `handleDropFTS` |
| 168 | +- `internal/api/server.go` — `/api/admin/drop_fts` route |
| 169 | +- `internal/mcp/tools.go` — description update + `clampTo24h` call in `toolSearchLogs` |
| 170 | +- `internal/config/config.go` — `LOG_FTS_ENABLED`, `VECTOR_INDEX_SNAPSHOT_PATH` (default `data/vectordb.snapshot`), `VECTOR_INDEX_SNAPSHOT_INTERVAL` (default `5m`) |
| 171 | +- `cmd/main.go` (or `internal/ui/ui.go` — wherever vectordb is constructed) — startup/shutdown wiring |
| 172 | +- `internal/telemetry/metrics.go` — 5 vectordb metrics |
| 173 | +- `CLAUDE.md` — FTS5 description, vectordb persistence note, config table additions |
| 174 | + |
| 175 | +## Acceptance criteria |
| 176 | + |
| 177 | +### Phase 0 (FTS5 + cap) |
| 178 | +1. `LOG_FTS_ENABLED=false` (default): new SQLite deploy has no `logs_fts` table; `search_logs` returns results via LIKE fallback. |
| 179 | +2. `LOG_FTS_ENABLED=true`: existing FTS5 path works unchanged (regression test). |
| 180 | +3. `POST /api/admin/drop_fts` reclaims expected bytes; returns 405 when flag is true. |
| 181 | +4. `toolSearchLogs` with no times → defaults to last 24h. |
| 182 | +5. start_time = 7d ago → clamped to now-24h. |
| 183 | +6. Window entirely outside cap → rejected with explicit error. |
| 184 | +7. HTTP `GET /api/logs?q=...` applies same cap. |
| 185 | +8. Updated tool description visible in MCP `tools/list`. |
| 186 | + |
| 187 | +### Phase 1-5 (vectordb persistence) |
| 188 | +1. Daemon restart → `find_similar_logs` returns useful results in <1s (vs hours pre-change). |
| 189 | +2. Tail replay correctness: seed N → snapshot → seed M → restart → exactly N+M entries, no duplicate `LogID`. |
| 190 | +3. `kill -9` between snapshots: previous snapshot loads, no `.tmp` left behind. |
| 191 | +4. CRC mismatch / wrong magic / version mismatch: warning logged, full rebuild kicks in. |
| 192 | +5. 5 new metrics present at `/metrics`. |
| 193 | +6. Manual: oteliq daemon shows `vectordb: loaded N entries from snapshot, replayed M from DB`. |
| 194 | + |
| 195 | +## Risks |
| 196 | + |
| 197 | +| Risk | Mitigation | |
| 198 | +|---|---| |
| 199 | +| Flag flipped mid-deployment leaves stale `logs_fts` | Stale table is harmless until 0b dropped. | |
| 200 | +| VACUUM on 4GB blocks writes 10-60min | Document; admin endpoint is opt-in, not automatic. | |
| 201 | +| 24h cap rejects legitimate historical queries | Explicit error; direct DB access remains for one-offs. | |
| 202 | +| LIKE fallback slower than expected at scale | Accepted trade-off. Escalate to v2 vectordb design if painful. | |
| 203 | +| Snapshot format breaks across Go versions | Magic + version + CRC → full rebuild on any decode error. Bump version on `LogVector` struct change. | |
| 204 | +| Cross-device rename loses atomicity | EXDEV detected, fall back to direct write. Constraint documented. | |
| 205 | + |
| 206 | +## Phasing for shipping |
| 207 | + |
| 208 | +5 PRs, in order: |
| 209 | + |
| 210 | +- **PR1**: Phase 0a + 0b + 0c (FTS5 disable + reclaim helper + 24h cap). Single PR — tightly related. |
| 211 | +- **PR2**: Phase 1 + 2 (snapshot encode/decode + Index integration). |
| 212 | +- **PR3**: Phase 3 (DB tail replay). |
| 213 | +- **PR4**: Phase 4 (snapshot loop + shutdown hook). |
| 214 | +- **PR5**: Phase 5 (metrics + docs). |
| 215 | + |
| 216 | +PR1 ships standalone. PR2-5 are sequential. Phase 0 can land before any vectordb work. |
| 217 | + |
| 218 | +## Verification |
| 219 | + |
| 220 | +```bash |
| 221 | +# Phase 0 |
| 222 | +go test ./internal/storage/... ./internal/mcp/... ./internal/api/... |
| 223 | + |
| 224 | +# Phase 1-5 |
| 225 | +go test ./internal/vectordb/... |
| 226 | + |
| 227 | +# Integration smoke |
| 228 | +go vet ./... |
| 229 | +go build -o /tmp/otelcontext . |
| 230 | +LOG_FTS_ENABLED=false /tmp/otelcontext & |
| 231 | +# Expect: "logs_fts: disabled (LOG_FTS_ENABLED=false)" on first start |
| 232 | +# After restart: "vectordb: loaded N entries from snapshot, replayed M from DB" |
| 233 | +``` |
| 234 | + |
| 235 | +## Resolved scope decisions |
| 236 | + |
| 237 | +- Snapshot path: flat `data/vectordb.snapshot` |
| 238 | +- zstd compression: not in v1 — ship uncompressed gob |
| 239 | +- Phase split: 5 PRs (above) |
| 240 | +- FTS5 default: off (`LOG_FTS_ENABLED=false`) |
| 241 | +- 24h cap interpretation: search window must overlap with last 24h; window entirely older is rejected, not silently emptied |
0 commit comments