Skip to content

fix(memory): stop layout divergence and serve the graph as map tiles - #586

Merged
ArgaFairuz merged 1 commit into
mainfrom
fix/memory-graph-layout-tiles
Nov 18, 2026
Merged

ArgaFairuz merged 1 commit into
mainfrom
fix/memory-graph-layout-tiles

Conversation

@ArgaFairuz

Copy link
Copy Markdown
Collaborator

Summary

Fixes the memory layout pass that diverged to NaN and failed on every run, replaces the binned "click to open" overview with a zoom-driven tiled map that shows every memory whenever the screen can draw it, and makes the map readable: spread-out lobes, density-aware node sizes, and a sparse edge backbone.

Why

Every layout pass since late October failed with NOT NULL constraint failed: graph_layout.x. Warm passes re-settled the map from its own previously spread result, inflating it ~1.4x per pass until coordinates reached 1e38 and overflowed. New memories were never placed, and the overview binned 5,171 memories into ~35 marks that had to be clicked open. After that fix, lobes still drew as solid blobs (neighbours ~1.6 px apart at fit, nodes 2–9 px) under 23k fully drawn edges.

Changes

  • Layout: warm passes place arrivals only (bounded local solver scaled to neighbour spacing, existing nodes fixed); cold passes on version change, divergence, or 20% growth, rigidly aligned onto the previous map (Procrustes) with community numbers preserved; non-finite rows never written; archived/superseded memories placed
  • Layout shape: per-lobe declumping to a minimum spacing and lobe compaction toward the centre; median nearest-neighbour distance 46 → 413 units on a real store; LAYOUT_VERSION 8 rebuilds once
  • Migration 089: graph_layout.degree and graph_layout_edges (derived edges the pass laid out on)
  • View: memory:graph returns meta (world, epoch, stamp, spacing, counts, extent); new memory:graph-tiles (screen-wide budget water-filled across tiles, edges deduped per request) and memory:locate
  • Canvas: tiles follow the camera; a store within budget loads once as one tile; complete tiles overzoom; scene applied as a diff with sigma's index settled; node size from map spacing with zoom-proportional growth; edge backbone at rest, full neighbourhood on hover/select, edges between search hits; selection/neighbours/hits always drawn; framing from the world box; restores the effects lost in perf(memory): persist the graph layout, bound the canvas, and let the shape follow the data #483
  • Store: tiles refetch only when the meta (epoch, stamp, placed count) changes or this client mutated something
  • Removed: region/bin view, "crowded spots" badge, "Back to everything", client layout worker

Test plan

  • bun run check, bun run lint
  • New backend/memory/layout.test.ts (13 tests, stable over repeated runs): exact stability across passes, arrivals move nothing and land at their partners' centre, divergence recovery, re-cold keeps colours, Procrustes alignment, spacing and stamp, tile budget/folding/stale/dedupe/filter/locate
  • bun run test: same pre-existing failures as a clean HEAD (shared ~/.clopen-test state in messages/git tests)
  • On a copy of a real 5,165-memory DB: cold pass ~15s, warm pass ~250ms moving 0 nodes, all memories individual at every zoom, rendered previews of fit and 6x zoom checked for overlap and edge clutter
  • Manual UI: open (centred), zoom/pan smoothness, hover/selection edges, search highlight and focus, inspector navigation, filters, archived, theme, live update, mobile sheet

Breaking changes

WebSocket: memory:graph now returns view meta instead of the full graph; tiles come from memory:graph-tiles. Only the bundled frontend consumes it.

Migration

Migration 089 runs automatically. The first layout pass after upgrade rebuilds the arrangement from scratch (seconds for a few thousand memories).

The Memory graph stopped updating and drew five thousand memories as
35 marks. Every background layout pass failed with "NOT NULL constraint
failed: graph_layout.x", so nothing recorded since late October was
ever placed, and the overview binned the whole store into a handful of
cells that had to be clicked open one at a time. Once that was fixed
the map drew its lobes as solid blobs under a grey mesh of lines.

Warm passes re-ran ForceAtlas2 from the stored positions and then
pushed the lobes apart again, but the stored positions were already
pushed apart and the simulation only pulled part of that back. On an
unchanged store each pass grew the map by about 1.4x; over weeks that
compounded to 1e38, where squaring a coordinate overflows to Infinity
and the pass produced NaN. The binned overview divided that
astronomically wide extent into a grid, which is why nearly every
memory fell into the same few cells.

Layout: a warm pass now moves nothing that is already placed. Arrivals
are dropped next to what they connect to and relaxed by a bounded local
solver scaled to the map's own neighbour spacing. The map is
re-arranged from scratch only on a version change, a diverged
arrangement, or once a fifth of the store arrived since the last cold
pass; the result is rigidly aligned onto the previous map and keeps its
community numbers, so colours and orientation hold. A cold pass now
spreads each lobe's dense core to a minimum spacing and packs the lobes
toward the centre without overlap, which took the median distance to a
nearest neighbour from 46 to 413 units on a real store. Non-finite rows
are never written. Archived and superseded memories get positions too.

View: the graph is served as map tiles over a fixed world square. The
server shares one screen-wide budget between the tiles on screen and
folds cells into clusters only when the screen cannot draw everything;
zooming in resolves them without clicks. A store that fits the budget
is fetched once as a single tile, and any complete tile stands in for
every deeper level. Derived edges, node degree and the neighbour
spacing are persisted by the pass (migration 089). The meta carries a
layout stamp, so reinforcement no longer refetches tiles.

Canvas: nodes are sized from the map's neighbour spacing and grow with
the zoom up to a ceiling, so they neither merge at the overview nor
shrink into the gaps when zoomed. At rest only each memory's strongest
links are drawn; hovering or selecting a memory shows all of its edges.
The selection, its connections and search hits are always drawn.
Framing is computed from the world box, because sigma never builds its
matrix while the graph is empty and the first framing happened then.
Sigma's index is settled after every structural change, which removes
the "can't be repaint" fallback. Restores the selection/search
highlight, theme repaint, hover from the results list and inset resize
effects lost in #483.
@ArgaFairuz
ArgaFairuz merged commit ca712fc into main Nov 18, 2026
1 check failed
@ArgaFairuz
ArgaFairuz deleted the fix/memory-graph-layout-tiles branch November 18, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant