fix(ansem): survive a slow memories table (site-down fix) - #404
Merged
Conversation
…e-flight /growth The memories table grows ~12-20k rows/day via the 24/7 live ingest. It outgrew its indexes (only single-column idx on owner_wallet / importance / created_at, nothing on source), so both DB-backed ansem routes started hitting Postgres statement timeout 57014: - /graph sorts the whole filtered set by importance to take the top 1000 - /growth ran ~10 SEQUENTIAL exact COUNTs per refresh /graph 500'd, the constellation rendered empty, and the page looked dead. /graph: cache the last good payload and serve it stale on any error/timeout; deadline the count (6s) and each node page (9s) so a stall degrades fast instead of holding the request for the full statement timeout. /growth: single-flight guard (concurrent visitors share one refresh instead of each starting a count storm), counts now run in parallel, stale-while- revalidate so a warm cache never waits on the DB, 8s deadline on a cold refresh falling back to zeros, TTL 15->30min, plus an ANSEM_GROWTH=false env kill switch. Does not touch config.ts. Index migration is the real fix and lands separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Post-index measurement: /graph is ~3.0-3.4s warm and ~19s cold, /growth ~4.6s. The original 9s page / 6s count deadlines would have fired on healthy-but-cold requests and served stale on every load. Raised to 15s page / 12s count / 12s growth — still far below the ~60s statement timeout they exist to guard. Also cache the graph payload for 120s: the node set is a 2000-row sample of a ~390k-row corpus and needs no per-request freshness, so almost every page load now serves from memory and never touches the database. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The outage
clude.io/ansemrendered empty. Server was healthy (/health200,/feedand/attestations~0.1s) — only the two routes that readmemorieshung:Root cause
memoriesgrows ~12–20k rows/day via the 24/7 live ingest and outgrew its indexes. Existing indexes are all single-column (owner_wallet,importance,created_at) with nothing onsource, so:/graphsorts the entire filtered set byimportance DESC, idto take the top 1000 (twice, paged)/growthran ~10 sequential exact COUNTs per refresh, with no coordination between concurrent callersBoth blew the statement timeout.
/graph500'd → empty constellation → "site is dead".This PR (application resilience)
/graph/growthANSEM_GROWTH=falseenv kill switch to shed all growth DB work instantlyDoes not touch
config.ts(drift trap avoided). Typecheck clean on the changed file.Still required — the actual fix
Composite indexes (run separately,
CONCURRENTLYsince ingest writes 24/7):🤖 Generated with Claude Code