feat(config): default STORE_MIN_SEVERITY=WARN for all drivers (don't persist INFO/DEBUG, keep analytics)#119
Merged
Merged
Conversation
Make WARN the global default for the store-severity gate (was "" globally, WARN only on SQLite). INFO/DEBUG logs now reach in-memory consumers (GraphRAG, Drain clustering, span/trace correlation) but are NOT persisted to the relational DB on any driver — only WARN/ERROR/FATAL grow the store. Analytics are preserved by construction: pipeline.process skips the DB row write for sub-WARN logs but still fires LogCallback over the FULL batch (TestPipeline_StoreMinSeverity_DropsBelowThresholdFromPersist), and the 60s GraphRAG refresh rebuilds topology from the spans table, not logs — so in-memory state/analytics don't depend on what's persisted. - config.go: StoreMinSeverity default "" -> "WARN"; remove the now-redundant SQLite-only override from applyDriverDefaults (WARN is global; SQLite no longer differs). - driver_defaults_test.go + CLAUDE.md updated to match. INGEST_MIN_SEVERITY stays INFO, so DEBUG is still dropped at the receiver (not analysed either). Set INGEST_MIN_SEVERITY=DEBUG to also feed DEBUG into in-memory analytics (raises in-memory event volume). Stats impact: error-rate/latency/p99/health/top-failing-services are trace/span-derived and unaffected; only the DB-derived TotalLogs count and log search reflect the WARN+ stored set (by design). Store-skips are metered via Pipeline.Stats().StoreFiltered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LDQVJrixs2nJoea67a8pEG
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make
STORE_MIN_SEVERITY=WARNthe global default (was""globally,WARNonly on SQLite). On every driver, INFO/DEBUG logs now reach in-memory consumers but are not persisted to the relational DB — only WARN/ERROR/FATAL grow the store.config.go:StoreMinSeveritydefault""→"WARN"; removed the now-redundant SQLite-only override fromapplyDriverDefaults(WARN is global; SQLite no longer differs).driver_defaults_test.go+CLAUDE.mdupdated to match.Why it doesn't lose state/analytics
The store gate is existence-preserving for analytics, verified two ways:
pipeline.processskips the DB row-write for sub-WARN logs but still firesLogCallbackover the full batch — proven byTestPipeline_StoreMinSeverity_DropsBelowThresholdFromPersist(INFO not persisted, yet itsLogCallbackfires).refresh.go) — so in-memory state/clustering doesn't depend on what logs are persisted.So GraphRAG, Drain clustering, anomaly detection, and span/trace correlation keep seeing INFO; the DB just doesn't store it.
Stats impact
TotalLogsdashboard count and log search (/api/logs,search_logs).Pipeline.Stats().StoreFiltered.This changes Postgres/MySQL/MSSQL too — they previously stored INFO by default and now won't. If you want INFO persisted on a given deployment, set
STORE_MIN_SEVERITY=INFO(or""). Easy to scope back to SQLite-only if preferred — say the word.INGEST_MIN_SEVERITYstaysINFO, so DEBUG is still dropped at the receiver (not analysed either). To also feed DEBUG into in-memory analytics, setINGEST_MIN_SEVERITY=DEBUG(raises in-memory event volume).Verification
go build/vet, gofmt clean, config tests (55), full suite 678 pass / 28 pkgs.🤖 Generated with Claude Code