feat(backends/memory): add the memory backend and its eval harness

[lIang70]

What

Adds backends/memory — the FlowCraft memory backend — plus backends/memory/eval, its own
module, which owns every measurement the backend has.

backends/memory assembles the memory.Assembly contract: message/document sources, the chat
and knowledge lines, fact/summary/document views, BM25 + entity + vector projections, fusion,
hydration, packing, maintenance and verification, over the workspace, sqlite and postgres
storage drivers.

backends/memory/eval is a separate Go module, so the backend itself never pulls a provider
driver. It carries the LoCoMo and LongMemEval loaders, the run harness (-answer, -prepare-all,
-resume, -answer-concurrency, -answer-style, -judge-style, -images), the official LoCoMo
token-F1 scorer, and the credentialed live probes.

Wiring. go.work gains both modules; the Makefile discovers nested backend modules and exposes
make memory-eval-probes; CI discovers nested go.mods and gains test-memory-postgres (real
Postgres 16, -race) and test-memory-windows lanes, both feeding ci-pass. make ci stays
offline: every credentialed probe self-skips without MEMORY_EVAL_LIVE=1.

Derivation generations and retention

Everything derived from a conversation belongs to a generation whose identity is the policy
digest, and replaced generations stay stored — that is what makes a rollback possible, and it is why
retention exists. Assembly.RetireGenerations(ctx, scope, conversationID, keep...) is the sweep: it
drops every generation a policy no longer keeps, across the facts, the summary bookmarks, the
projection lanes, and the derivation progress that would otherwise resume in the middle of them, in
one call.

The sweep is ordered, and now guarded, because each derived view addresses its stored state
differently:

• The progress goes with the generation. A watermark is where the next pass under that policy
  resumes, so a generation whose facts are retired while its cursor stays at the end of the stream is
  one a rollback resumes in the middle of: it derives the tail, publishes a generation missing the
  rest, and the commits the cursor passed are never derived again. In the fixture that reproduced it,
  the sweep retired RetireResult{Facts:1, Summaries:1, LaneEntries:1}, the rollback derived 1 commit
  and the conversation served 1 fact of the 2 it had. RetireResult.Watermarks reports the cursors
  the sweep retired, and they go before anything else: the cursor is the one thing a sweep removes
  that is not derived state, so an interruption leaves facts without a cursor — which re-deriving
  reproduces, since the derived addresses are content derived — rather than the state the sweep
  exists to prevent, facts gone under a cursor that still points past them. The published
  generation's cursor is refused (Processor.RetireProgress): an ordinary pass under a policy whose
  generation is visible would derive the stream a second time, and what a re-derivation produces is
  the model's, so the visible generation would grow beside itself instead of converging. A sweep
  never names it.

• The lanes go next. A lane entry is addressed by the id of the fact it projects, and the
  converge walks the stored generations, so once a generation's facts are retired those addresses
  can never be enumerated again: the lane keeps offering a candidate the read path cannot hydrate,
  and Verify cannot see it — it replays the projection chain, which recomputes the digests the
  leaked entries were chained with. The generation that reaches that state is one whose pass failed
  after projecting.

• The bookmarks go before the facts, so an interrupted sweep leaves unreachable facts, which the
  next sweep collects, instead of a bookmark to facts that no longer exist.

• Derivation has to be quiesced, and the sweep enforces it. It refuses while a pass over the
  conversation is running, or while this worker has derived its own generation without publishing
  it, and returns worker.ErrDerivationUnsettled; Assembly.DerivationState reports the same state
  before the fact. A pass publishes the generation it owns whether or not the sweep emptied it, and
  it resumes after its own watermark, so a sweep beside it costs the commits that generation had
  already covered — in the fixture that reproduces it, a sweep that retired the generation the
  worker owed (Facts:1, LaneEntries:1) left the republished generation serving 1 of the 2 facts
  the conversation had derived. Both refusals are retryable and both mean the sweep retired nothing:
  derive again (RunOnce is synchronous) and sweep after the pass.

• A rollback into a swept generation derives it again. Retirement takes the generation's cursor
  with it, so the pass starts at the head of the stream instead of after the commits whose facts the
  sweep removed, and compaction publishes the manifest for the generation it summarises, so the
  summaries come back with the facts rather than leaving the branch where the sweep left it.

• A pass that derives nothing still moves the views. The summary branch is republished with the
  generation the pass is about to publish, so a rollback — which finds its watermark already at the
  end of the stream — serves its own summaries rather than the replaced generation's.

• Documents are not generation scoped, so they are not swept with one. A chunk id comes from the
  document's provenance and text, not from the policy that derived it, so deriving a document again
  under an earlier policy reproduces the build it published then — there is nothing a rollback would
  need to find. What a document does leave behind is a superseded build: a new revision, a changed
  chunking, or a publish that failed after writing its chunks and before moving the pointer. Reads
  resolve only the active build, and the per-conversation sweep never touches the view, so the
  maintenance pass reclaims the superseded builds at scope granularity instead — Assembly.Maintain
  reports Plan.ReclaimedChunks and keeps the active build of every document of the scope. In the
  fixture, two revisions of one document left 4 superseded chunk records: the first pass removed all
  4, the second removed none, and the active build still served.

The full contract, including what a sweep keeps when the views disagree and why per-view sweeps stay
available for tooling, is in backends/memory/README.md.

Measurements

LoCoMo, 1540 questions, categories 1–4. Category 5 (adversarial) is excluded by design and printed
as skipped_adversarial=446 on every run. Config: -samples 10 -max-items 30 -max-tokens 6144 -images native -answer -answer-style short -prepare-all -answer-concurrency 4.

metric	value
official token-F1 (categories 1 / 2 / 3 / 4)	59.85% (43.6 / 63.7 / 36.4 / 66.5)
questions whose full evidence was recalled	1380/1540 = 89.6%
turn-level evidence recall	2113/2354 = 89.8%
answer accepted by containment grading (no judge model)	544/1540 = 35.3%
strict judge, long-answer configuration	78.4%
locomo judge, long-answer configuration	90.1%
retrieval latency	535 ms per question (10.4 s before)
full run, end to end	~29 min

The two judge rows are a different protocol from everything else in this
table and from the answer-model comparison below: they graded the long-answer
prompt (answer-policy-v2) with judge-v1, while the short-answer runs use
answer-short-v1 with judge-v2. A judge scores terse answers harder (there is
less text for it to accept), and judge-v2 records an ungraded verdict instead
of guessing, so 78.4%/90.1% and 67.8%/85.9% are not comparable numbers -- only
the official token-F1 is, and it is 59.85% in both.

Containment grading is a string-containment grader: it accepts an answer only when
the gold text appears in it. A paraphrasing model scores low on it without being
wrong (Counseling and mental health vs a gold Counseling and mental health is
accepted; Counseling or mental health. is not), which is why the cross-model
number below is the token-F1 one.

Answer-model comparison, one frozen derivation

-skip-derive answers from the derivation the workspace already holds, so both
runs retrieve the identical context and a difference is the answering model's.
Both arms answer short and grade themselves with -judge-style both:

answering + judging model	official token-F1 (1 / 2 / 3 / 4)	strict judge	locomo judge	evidence recall	wall
deepseek/deepseek-flash	59.85% (42.5 / 63.9 / 35.0 / 67.0)	67.79%	85.91%	2107/2354 = 89.5%	24m47s
openai/gpt-4o-mini	51.32% (38.4 / 60.3 / 15.4 / 56.3)	56.17%	82.08%	2102/2354 = 89.3%	17m54s

Evidence recall differs by 0.2pp, so the memory and the recalled context are the
same in both arms and the gap belongs to the answering model: 8.5pp on the
official metric, 11.6pp on the strict judge, concentrated in category 1 and
category 3. gpt-4o-mini answers ~28% faster and emits far fewer output tokens
(15k against 971k for the answer pass), which is the trade it offers.

An earlier pass on the previous derivation (before chat.AlgorithmVersion
1.3.0) measured the same ordering: 60.02% for deepseek-flash answer-only
against 52.56% for gpt-4o-mini, both with containment grading instead of a
judge.

Reranking ships off: after the tail-scoring fix, the A/B moved strict by +0.7pp and lenient by
−1.1pp, both inside the noise band the README documents (±1 question, <1pp — same-config reruns of
the full set differ by 2 questions). retrieval.decompose was removed: it measured net negative.
The context budget was not relaxed either; 45 items was measured and rejected.

Model configuration (eval/deploy.yaml)

• deepseek (deepseek-flash) extracts facts; deepseek/deepseek-v4-pro is the other model the
  endpoint serves, and any other name is silently substituted — TestConfiguredModelsExistInCatalog
  now fails a run that names one.
• bytedance (doubao-embedding-vision) embeds: one text/image item per request, pinned to the
  dated snapshot in the profile.
• openai (gpt-4o-mini) is declared for alternative answer runs. It is multimodal, so its
  capabilities declare image input; it rides the responses surface (the chat surface rejects the
  json_schema format the fact extractor asks for) and states wire.store: false explicitly.
  The same live test now checks the OpenAI catalog too and fails on a model whose shutdown_date
  is set, rather than discovering a retirement mid-run.

Reproduce

cd backends/memory/eval
cp .env.example ../../../.env   # DEEPSEEK_API_KEY, ARK_API_KEY (OPENAI_API_KEY for the openai provider)
go run ./cmd/memory-eval -deploy deploy.yaml -env-file ../../../.env \
  -samples 10 -max-items 30 -max-tokens 6144 -images native \
  -answer -answer-style short -prepare-all -answer-concurrency 4 \
  -judge-style none -out /tmp/locomo10.json
MEMORY_EVAL_REPORT=/tmp/locomo10.json go test ./official/ -run TestScoreStoredReport -v

[lIang70]

Posting my own review notes on this PR so reviewers have the full list up front.
I went back through the new modules (storage drivers, sources, lines, views,
retrieval, worker, eval harness) and did targeted checks against the code.
Quick summary: 1 privacy bug and 1 correctness bug against a stated invariant
(items 1-2), 2 driver-contract inconsistencies (3-4), then a batch of medium
items and nits. Details below; everything is verifiable at commit 51131d23
(the current head).

One note up front: I did not re-run the live eval lanes in this pass (they need
MEMORY_EVAL_LIVE=1 plus provider keys), so the reported measurements are not
reproduced here - this is code review only. And for a diff this size I focused
on the new modules; the Makefile/go.work/CI wiring I only skimmed.

The areas I am most confident in are the durability semantics (log-then-KV,
publish-forward/rollback on crash, idempotent commit replay), the shared
conformance suite, content-addressed ids, and the retrieve/pack budget
discipline. The items below are what I want resolved or consciously accepted
before merge.

Should fix

1. retrieval/provider.go:722-742 - the source-quote cache is keyed without
scope, so quotes can leak across scopes.
quoteText caches message text in provider.quoteTexts under sourceID only
(the id is conversationID + "/" + messageID, built at provider.go:646 and
worker/processor.go:686). The cached value is read under the request's scope
(provider.Messages.Get(ctx, scope, ...) at provider.go:732), but the key
omits it, and one Provider is shared by every scope an assembly serves. If two
scopes ever use the same conversation id (caller-supplied; nothing enforces
cross-scope uniqueness), the second scope's facts get the first scope's raw
message text folded into their content as "Source turn: ...". Conversation ids
like conv-1 are exactly the value that collides across tenants. The flag is
off by default but on in the measurement config (eval/deploy.yaml:153 sets
source_quotes: 3). Fix: include scope.HardPartitionKey()
(core/memory/scope.go:41-43, \x00 separator) in the key, and consider
bounding the map - it is process-lifetime and unbounded today. The existing test
only exercises one scope (source_quotes_test.go).

2. retrieval/pack/pack.go:165-183 - the newest recent turn can still be
dropped, contrary to the stated invariant.
provider.go:292-296 truncates the newest turn to recentTokenLimit "so the
packer can still admit it", and context_recent.go:101 says "The newest turn is
bounded, never dropped". But within the recent class, items are admitted
oldest-first: prepared is sorted by lessItem, which orders recent items by
ascending Sequence (pack.go:296+), and the first pass walks that order
(pack.go:165-172). The newest item is admitted last, and the lending pass only
adds - it never rebalances (pack.go:175-183). Concrete case with
budget.max_tokens = 100: older recent turns m1 = 55 tokens, m2 = 30 tokens get
admitted (m1 in the class pass, m2 via lending); the newest turn, truncated to
min(recent lane, 100), then fails both the recent class cap (55 + 100 > 60)
and the global cap (85 + 100 > 100) and is dropped. The existing
TestProviderBoundsOversizedNewestTurn pins only the single-turn case; adding
one older sibling turn makes the newest disappear. Fix options: reserve a slot
and budget for the newest item, or select the recent class in descending
sequence and rely on the final re-sort (pack.go:187-190) to restore reading
order - plus extend that test with an older turn.

3. storage/sqlite/store.go:301-326 and storage/postgres/store.go:286-311 -
ListStreams("") returns nothing on SQL drivers, everything on the workspace
driver.
With an empty prefix, likePrefix("") yields "/%" (sqlite :641-644,
postgres :616-619), which matches no stream, so the post-filter
if prefix == "" || ... (sqlite :321, postgres :306) is dead code - it is
written as if "" meant "all streams", and the workspace adapter implements
exactly that (workspace_log.go:246+ walks from the log root; pinned by
workspace_log_test.go:159-161). The interface comment (storage/log.go:62-65)
does not pin the empty-prefix case and the shared conformance suite has no
empty-prefix subtest (internal/storagetest/storagetest.go:28-30), so the
drivers disagree silently. The only production caller uses a non-empty
partition prefix (sources/message/store_impl.go:245), which is why it has not
bitten yet. Pick the contract ("" = all streams looks intended), make the
SQL drivers honor it, and add a conformance subtest.

4. storage_drivers.go:54-56,86 - omitting schema in a Postgres config opens
two pools where one is promised.
postgresKey builds DSN + "\x00" + config.Schema from the raw config, but
postgresstore.Open normalizes an empty schema to DefaultSchema, and
DSNKey() returns the normalized pair (storage/postgres/store.go:45-49,86-90).
So for a config without schema, the dedupe check at storage_drivers.go:86
can never match and log + KV get separate pools and two migrations -
contradicting README.md:102 ("KV sharing one DSN+schema share one pool").
Fix: compare against the normalized key (e.g. default the schema in
postgresKey via postgresstore.DefaultSchema).

Worth a decision / lower priority

5. eval/official/f1.go:14-35 - the scorer's normalization is not identical to
the reference harness.
The reference (snap-research/locomo, task_eval/evaluation.py) strips commas,
then deletes ASCII punctuation membership-style
("".join(ch for ch in text if ch not in string.punctuation)), then stems with
NLTK's PorterStemmer. This implementation replaces every Unicode
punctuation/symbol rune with a space (f1.go:18-24) and stems with
reiver/go-porterstemmer. Observable differences: Caroline's -> reference
carolines, ours caroline s; gpt-4o -> gpt4o vs gpt 4o; 1,000 ->
1000 vs 1 000; non-ASCII punctuation (curly quotes, dashes) is kept by the
reference but split into separate tokens here. Since the package exists so numbers
can sit "next to published ones" (report_test.go:16-18), I would either align
both normalization details exactly (comma strip + ASCII-only deletion) and
cross-check the stemmer on a small fixture of tricky strings, or soften the
"same ruler" wording and document the accepted divergence.

6. eval/cmd/memory-eval/main.go:193-225 - -skip-derive does not fingerprint
the frozen memory it reads.
The fingerprint pins mode (derive=reused), deploy document, models and
budgets, but not which derivation the workspace holds. The README pitches the
mode as "compare two answering models against one frozen memory"
(eval/README.md:149-157); nothing stops -resume/-baseline from mixing runs
whose "frozen" memory actually changed between them (another derive pass, a
Maintain run, a workspace swap). Cheap fix: fold a workspace-state digest into
the fingerprint (watermarks + record counters from Diagnostics, or a hash over
the facts/summary projection manifests).

7. factory.go:171-174 vs :230-233 - swapping in a deriver via WithDeriver
does not change the policy digest.
policyDigest covers settings only (policyInput, :507-537), while
buildDeriver returns the caller's deriver unconditionally (:230-233).
Watermarks are keyed by that digest (worker/checkpoint.go:31, used from
worker/processor.go:310,351), so after a host replaces the deriver, existing
scopes are considered up to date and are never re-derived. Either let
WithDeriver carry a policy version that folds into the digest, or document the
re-derive procedure.

8. lines/chat/fact_extractor.go:665-671 - a missing timestamp becomes Unix
epoch, not "unknown".
artifactEventTime returns time.Unix(0, 0) when the artifact carries no
event_time, and withinFiveMinutes only guards IsZero (:726-732), so
epoch == epoch makes untimestamped facts look "within 5 minutes" of each other;
maintain.recencyKey (maintain/plan.go:166-170) also prefers the epoch over
CreatedAt, landing those facts at the decay floor. The extractor prompt says
the opposite: "If no timestamp is available anywhere, omit the date rather than
fabricating one" (prompts/fact_system.tmpl:15-17). The built-in worker always
sets event_time (worker/processor.go:620), so this is reachable through the
public Deriver contract with host-supplied artifacts. Fix: return a zero
time.Time / omit the metadata key and let consumers default it.

9. sources/importer/importer.go:134 - AllowLoopback: true disables the
whole address guard.
The flag's documentation says "permits loopback targets (tests and local
development)" (:28-29), but the condition at :134 skips blockedAddress
entirely, unlocking RFC1918, CGNAT, link-local (including 169.254.169.254
metadata) and reserved ranges for a caller that only wanted local dev. Only
tests use it today - good moment to split the policy (AllowLoopback vs an
explicit AllowPrivateNetworks), or at minimum keep metadata/link-local blocked
unconditionally.

10. assembly.go:102-183 - small lifecycle gaps.
cancel/done are written inside Wire and read in Close with no
synchronization (:102-125 vs :127-143), so concurrent Wire/Close is a data
race; and Close-before-Wire silently does nothing, after which Wire can still
start the runner (nothing can stop it) and use already-closed pools. In
runLoop (:175-177), RunOnce's error is dropped with a comment saying it is
available via Diagnostics/LastError - true for processor failures
(worker/processor.go:166-171 records them), but a catalog.List failure
(:150-155) returns before the processor and is never recorded. Suggest a
closed-state guard plus recording that error via processor.RecordError.

11. lines/chat/fact_extractor.go:636-641 - the "rich" strategy's schema
requires fields the prompt never describes.
For StrategyRich, schemaFor requires predicate and temporal_detail,
while the only prompt template lists exactly three keys ("text", "entities",
"event_time"; prompts/fact_system.tmpl:40-43). On strict providers the model
must invent two fields; on the json_object fallback they silently arrive
empty; and nothing downstream consumes them beyond persisting them
(worker/processor.go:649, views/fact/fact_store.go:130,
views/fact/types.go:32-33). Either document them in the template (with
examples) and use them, or drop them from the rich schema.

12. Release plumbing - decide the intended story.
backends/memory requires modernc.org/sqlite v1.59.0 while
backends/checkpoint pins v1.56.0 (backends/checkpoint/go.mod:7): under the
workspace, checkpoint's CI lane compiles and tests against 1.59.0, so what it
ships (1.56.0) is not what CI validated. Separately, eval/go.mod:72-74
promises "drop this replace and pin the released version when backends/memory is
published", but tools/releasegate/main.go:21 only knows core - the publish
path is not wired. Either extend releasegate (and add a pinned/GOWORK=off lane
like examples/forge gets in ci.yml:106-107) for the new modules, or convert
that comment into a tracked issue and state explicitly that backends/memory is
workspace-only for now.

Minor / nits

• retrieval/rerank/rerank.go:169-179: the comment concludes "so v1 stays" but
  AlgorithmVersion = "rerank-v2", and that string is a fingerprint input
  (eval/cmd/memory-eval/main.go:214). Reconcile so run labels are
  unambiguous.
• sources/message/store_impl.go:296-305: readRecords guards with
  uint64(len(events)) != last-first+1, which wraps to 0 for an inverted range;
  a corrupt commit marker with last < first passes and then records[0]
  panics (:133, :233, :291). Add a last < first / empty check.
• views/summary/store.go:380-384 (+ :126): equivalentRecord ignores
  GenerationID, so re-Adding identical content under a new generation
  returns the old record, with the old GenerationID. The active path is fine,
  but generation-filtered reads will disagree with what a caller just wrote -
  decide and document.
• views/summary/search.go:23-48: LoadActive and ListActive take the lock
  separately; a publish in between makes the candidates carry the older
  manifest's generation id (and a filtered ListActive can return empty). One
  locked read would make the lane consistent.
• views/fact/fact_store.go:676-679: the first condition in stateContains can
  never change the outcome (the final sourcesContain dominates); as written it
  reads like a deliberate guard that is not. Collapse it or comment the intent.
• Live-only tests: eval/cmd/memory-eval/context_size_live_test.go:143,148
  update capped/tokens outside mu, a data race under -race when
  MEMORY_EVAL_LATENCY_WORKERS>1; rejudge_live_test.go:180-200 never
  increments the judge-v1 lenient counter, so that log line always reads
  lenient=0.0000.

What I checked and found fine (non-exhaustive)

Storage: workspace log write/publish ordering with crash-forward/rollback
recovery tests, idempotent commit replay and cursor semantics; workspace KV
compare-and-swap; the shared conformance suite design. Sources: document/dataset
log-then-KV with repair, version/tombstone monotonicity; SSRF resolve-then-dial
ordering (before the AllowLoopback caveat). Retrieval: hydrate/pack
determinism, dedupe by address, total budget never exceeded, budget clamping;
fusion and scorer unit tests. Views: fact merge/aggregate replay and digest
determinism; summary base + delta catalog algebra and digest verification. Eval:
fingerprint coverage of model/budget changes (item 6 is about the memory state
itself, not config coverage). The READMEs are unusually thorough.

Limits

I did not re-run the live lanes in this pass, so the reported numbers are not
independently reproduced; everything above is code-level. The "found fine" list
names where I looked closely, not a proof of absence. Unless reviewers prefer
otherwise, my plan is to file the should-fix and decision items as issues and
fold the two P1 fixes into this branch.