Skip to content

feat(dreamer): governed automatic consolidation from model proposals to usable memory #238

Open
Open
feat(dreamer): governed automatic consolidation from model proposals to usable memory#238

Description

@joshyorko
joshyorko
opened on Sep 10, 2026

Governed Dreamer consolidation: automatically adopt useful, evidence-backed memory

Outcome — useful learning, not permanent preview

Build governed automatic consolidation: after an operator enables a scoped learning policy once, useful Dreamer output becomes durable, correctly attributed memory and improves a later conversation without asking the user to approve each memory.

“Assimilation” describes the adoption step, but this is an extension of Dreamer and MemoryD, not a new product, agent runtime, scheduler, graph database, or memory service.

The complete journey is:

permitted evidence -> bounded Dreamer proposal -> durable candidate batch
 -> source/semantic validation -> policy decision -> transactional adoption
 -> current recall / attributed learned beliefs -> correction and feedback

The user should experience better continuity, fewer repeated explanations, useful connections, and straightforward correction. Neither “the model generated a dream” nor “the database has more rows” is success. An implementation that defers every candidate also fails.

This issue authorizes the implementation described below, not live automatic activation, a production corpus rewrite, external actions, or resumption of unrelated paused goals. Package an activation path that needs one scope/policy decision, not endless per-candidate confirmation. Complete the automatic path in isolated acceptance before requesting any necessary production rollout decision.

1. Verified baseline and precise gap

Source review: 2026-09-10, master@6ac53f1197132e4bfddfa2a2decd4862dab5c466. Re-read current heads and existing worker/PR ownership before editing. This issue's author reviewed source and reported PR evidence; it is not a claim that tests were executed here or that a consumer's live installation was inspected.

Read these exact seams, not just the README

All links below are pinned to the reviewed source.

File / symbol Observed behavior and required implication
src/service.rs: run_dream_job, augment_model_dream_preview Kind is restricted to dream_preview; generation runs deterministic preview and then the model. Model-backed candidates are made ineligible for apply. Full output is returned, while job/run persistence is principally metadata/audit. Store the actual immutable proposal and disposition before treating it as recoverable.
Same file: scheduled_dream Command mode rejects automatic_apply; the older HTTP path may call promote_provider_observations. A successful capped preview can advance the scheduled watermark to now. Provider transport must not determine memory-trust policy, and a truncated batch must not skip its unprocessed tail.
Same file: patch_preview, patch_apply, patch_rollback patch_apply regenerates a deterministic preview, compares run ID, then reruns deterministic apply. It does not apply the stored Luna proposal. Implement exact-proposal application and guarded undo; do not claim the existing command already does it.
Same file: dream_provider_context, provider_candidate_from_value Model input currently reduces source entries to ID/kind/content, dropping useful actor/time information. Candidate checks establish allowed reference membership, not semantic support. Duplicate cited IDs can also overstate counts. Preserve source meaning and compute distinct evidence roots server-side.
Same file: promote_provider_observations The legacy helper screens/classifies, copies evidence IDs, marks accepted, writes, and can archive. Do not merely call it from command mode or write proposals through /conclusions to evade the new gate.
src/dream.rs: build_evidence_window, source-stream builders, evidence scoring, run Reuse evidence/temporal machinery. Source streams each receive a limit, so enforce a combined budget. stream_from_conclusions uses target as actor: a conclusion about a user is not necessarily written or confirmed by that user. Legacy scoring weights are not semantic proof or calibrated probabilities.
src/provider.rs, native companion Reuse typed inference and native credential resolution. Count complete prompt/schema overhead, validation calls and actual effective model/adapter identity. No agent/tool executor is needed for synthesis or validation.
src/store.rs, 0011_dream_jobs.sql, src/protocol.rs Reuse the canonical SQLite/evidence/job store and versioned protocol. dream_jobs does not contain a complete proposal lifecycle. A run/input hash alone is not an immutable model-output identity.
src/recall.rs, native Hermes provider under adapters/hermes-memory-provider/ Actual consumer context must preserve attribution/currentness and expose useful learned material. Storage success without usable recall does not close this issue.
docs/async-dreamer-v2.md, docs/dreamer-native-provider.md, tests/dream_jobs.rs, tests/dream_command_scheduler.rs Update documentation and add acceptance for the new mode, while retaining real preview/no-promotion tests for the existing mode. Do not weaken those tests globally.

Post-merge #237 findings to reproduce before unattended adoption

The post-merge review reports the following. Reproduce against the current base; reuse a current fix owner when present. A review comment is not a completed fix.

  1. resolve_dream_provider: configured command jobs can ignore provider_enabled=false on direct invocation.
  2. First scheduled command run passes since=None, which the job replaces with the manual apply watermark. An explicit empty scheduled cutoff must not mean “inherit another cursor.”
  3. Caller-supplied provider name can mislabel the fixed command's provenance; command runs also inherit http-json-v1 as adapter version.
  4. Synchronous scheduled inference can block a Tokio worker and therefore foreground recall/shutdown, especially with one runtime worker.
  5. Command-reported/estimated monetary costs are not accumulated using the same rolling daily check as the HTTP Provider adapter.
  6. Reported input tokens can exceed the input-only estimate once the companion adds schema/system text.
  7. Status can report a larger output-byte limit than the minimum actually imposed by scheduler/runner/companion; file existence alone also does not establish executable readiness.
  8. The provider schema can request more candidates than the slots remaining after deterministic results, paying for output that cannot be retained.

These become Packet A regressions and integration gates, not a reason to postpone all independent contract/fixture work. A genuine repair already landed elsewhere should be referenced and tested, not duplicated.

2. Research decisions — borrow behavior, not another platform

Primary sources were inspected on 2026-09-10. Product documentation is evidence of a design/API, not proof of superiority, safety, or a benchmark result on this repository.

Source Idea to adopt here Boundary
Honcho Dreaming Separate explicit knowledge maintenance from inferred patterns; run scoped incremental consolidation with deduplicated pending/running work. Honcho labels dreaming experimental. Do not copy its numeric scheduling thresholds, personality profiling, or deletion behavior. Two derived conclusions are not necessarily two independent sources.
Mem0 Add, History Minimal capture UX, bounded prior conversation context for references, distinguish PENDING from completed, and inspectable changes. Current Add documentation describes additive/ADD-only storage; do not assert the older ADD/UPDATE/DELETE description applies universally. History alone is not safe rollback.
Mem0 Dream configuration, synthesized-memory sources A clear learning-mode control and a direct path from learned memory back to contributing sources. Adopt the user-facing contract, not hosted entitlements or claims about undocumented synthesis internals.
Hindsight observations Durable derived beliefs with source quotes; reconcile near-duplicates; invalidate/refine derived views when evidence changes. Similarity only selects comparisons. Preserve numbers, negation, names, language, scope and time. No mandatory embeddings backend.
Hindsight mental models Small useful derived views can accelerate later context, with freshness and source fallback. Use existing MemoryD cards/packs; do not create a second personality store or make a cached summary authoritative.
Graphiti Separate source/valid/ingestion time and preserve superseded history with source-linked relationships. Reuse MemoryD temporal/semantic primitives; no new graph database or ontology mission in this issue.
Honcho duplicate-observation report #729 Add repeated-dream/paraphrase tests: a memory system can have deduplication and still fill context with redundant beliefs. This is a reported failure on the reporter's revision, not a blanket claim about current Honcho. Do not adopt lowercasing/date-stripping as immutable identity rules.

Key design correction: deterministic code can enforce scope, source identity, revision, operation and budget rules. It cannot establish arbitrary natural-language entailment merely because a cited ID exists. Use bounded semantic checking where necessary and report its uncertainty; another model agreeing is not independent evidence.

3. Product contract and policy

One setup decision, routine autonomous learning afterward

Add a versioned, operator-owned policy selecting off, preview, or automatic for declared scopes. Existing installations do not silently opt in during upgrade. Preview remains non-adopting. In automatic mode, eligible memories are adopted without per-candidate confirmation.

A policy states allowed claim classes, source classes, scopes/perspectives, operations, budgets, retention, currentness rules and the semantic-validation route. Model-generated text and per-job payloads cannot enable automatic mode, widen a grant, select a different policy, or change provider credentials.

A directly stated ordinary preference can be useful after one clear statement. Do not require users to repeat themselves three times or say “remember this” for every benign fact. Conversely, one ambiguous joke, quoted example or hypothesis is not an enduring preference.

The initial automatic policy must support both:

  • Attributed statements: what a user explicitly said, or what a verified source actually established, with the correct subject, context and known time basis.
  • Learned beliefs: useful, non-sensitive inferences supported by the evidence, durably retained and recallable as inferred, not falsely described as user-confirmed or independently verified facts.

Deferred proposals are different from admitted inferred beliefs. Weak/contradictory proposals may wait quietly for new evidence; uncertainty must not force every useful inference into permanent quarantine.

Closed dispositions and permitted effects

Disposition Meaning
adopt_statement Persist an accurately attributed, eligible statement. A belief about a person's preference remains scoped to that person/context.
adopt_inference Persist an eligible derived belief with its supporting/counterevidence and explicit inferred status.
reinforce Link genuinely new independent support to an existing equivalent claim, without a duplicate claim or fabricated freshness.
supersede Replace the current interpretation using supported changed evidence; retain permitted historical source meaning and invalidate affected current views.
no_change The proposal is already represented; repeating a derivation supplies no new primary evidence.
defer Record a specific reason and what new evidence could change it; no repeated user approval question.
reject Unsupported, prohibited, irrelevant or malformed candidate, with a content-safe reason.

All effectful dispositions go through one server-owned application boundary. No automatic hard deletion, permission change, external action, arbitrary procedure execution, profile identity rewrite or broad entity merge. Sensitive personal inference is excluded by the initial policy; ordinary permitted personal memory is not excluded merely for being personal.

Explicit unambiguous user corrections must not require approving the same correction again. Their source/target linkage and supported correction operation provide the authority for that bounded update. Genuine unresolved conflicts should be qualified/deferred or yield one useful clarification—not silently resolved by whichever text arrived last.

recall_not_authority remains a distinction between context and execution authority. It is not a prohibition on automatically learning. It also does not make a wrong memory harmless: bad memory can bias decisions, so source fidelity, correction and evaluation still matter.

Semantic checking without an expensive review bureaucracy

  1. Construct a bounded source packet including actual speaker/producer, subject/perspective, source kind, source content, known times, lineage roots, current disposition and relevant counterevidence. Do not drop actor/time as the current model packet does.
  2. Ask the proposer for atomic claims, exact source references/short supporting spans, claim kind and qualifiers. Do not ask it to choose its own permissions or trusted source class.
  3. Resolve references and exact supporting spans against retained sources server-side. Detect quoting, negation, conditional language and scope changes as semantic questions, not just substring matches.
  4. For generated paraphrases, inferred patterns, semantic equivalence and nontrivial supersession, perform one bounded evidence-focused validation pass per batch using the configured affordable inference route. Supply the candidate, actual supporting material and relevant contrary/current claims; not the proposer's private reasoning. A deterministic exact-source rendering path may skip this pass only when it adds no semantic claim beyond the attributed source representation.
  5. The validator returns a closed verdict (supported, partially_supported, contradicted, insufficient) plus source/qualifier references and a concise explanation. It cannot mutate memory or call tools. Parse/validate it as untrusted input. Only the applicable policy can admit a candidate, with model fallibility still recorded.
  6. Partial support cannot adopt the whole compound statement. Split/revise into supported atomic claims, bind a new digest and revalidate within budget, or defer; never silently trim a reviewed artifact after approval.

For the initial inference lane require at least two distinct primary evidence roots, compatible scope/context and a supported validation result. This is an eligibility heuristic, not a probability guarantee. Direct statements do not need two roots. Re-evaluate this policy only through explicit versioned evaluation; do not tune it mid-run to make a failing candidate pass.

Source authorship/adoption comes from trusted host capture and retained provenance—not target=user, endpoint strength, a model-supplied actor, confidence 0.99, or repeated assistant summaries. A verified engineering result can be primary evidence for that result, not proof of a personal preference.

4. Durable proposals, exact identity and safe commit

Candidate lifecycle

Use the canonical SQLite store. Reuse existing job/run/patch/evidence primitives; add a narrowly owned candidate-batch/disposition representation where their current schemas are insufficient. Do not implement a separate queue service.

Persist before acknowledging recoverable generation:

  • scope/principal/perspective and policy ID/version/digest;
  • source snapshot identity, exact references/digests/revisions and bounded coverage cursor;
  • proposer and validator effective model/provider/adapter versions, input/output digests and usage;
  • immutable candidate set with per-candidate IDs, qualifiers, proposed disposition and evidence roots;
  • processing state, decision, failure/defer reason, source/policy preconditions and eventual receipt IDs.

Suggested processing states: generated -> validated -> adopted | no_change | deferred | rejected | stale | expired. Operational failure is separate from rejecting a belief. Bind source references and candidate output, not a mutable free-text explanation.

Input identity is not output identity: two model responses to the same prompt may differ. Include a canonical complete proposal-output digest in the immutable batch identity; an exact stored proposal retry returns the same receipt, while another response is a distinct candidate set. Duplicate IDs with different payloads conflict.

Repeated evidence IDs and copied summaries must collapse to their known primary lineage roots for support counting. Unknown lineage does not establish independence. Similarity can propose a comparison; it cannot establish immutable identity or erase meaningful differences. Preserve case-sensitive commands, whitespace, units, quantities, negations, named entities and validity intervals; coordinate exact identity with #231.

Deferred/rejected candidates retain a compact reconsideration fingerprint. Do not re-propose the same rejected idea every cycle absent materially new source/policy evidence. Enforce count/age/byte limits without removing the minimal receipt or suppression information needed for correctness. No secret/raw-rejected payload in errors, logs or audit.

Reference application algorithm

The following names are a proposed new contract, not existing APIs. Freeze equivalent names once in Packet B and pass that contract to every worker.

prepare_consolidation(scope, policy_ref, source_cursor, budget)
    -> immutable ConsolidationBatch

decide_consolidation(batch_ref, bounded_validation_result)
    -> persisted ConsolidationDecision[]

apply_consolidation(batch_ref, decision_digest, expected_policy_revision)
    -> ConsolidationReceipt

Any exposed mutation request references the stored batch/decision, not caller-supplied replacement memory text, SQL, arbitrary record IDs or an approve_all flag. CLI/HTTP/native consumers share the same service and configured capability checks; this issue grants no remote write access or broader client permissions. Policy is loaded from the configured authority at commit time.

resolve permitted scope and exact stored batch
load current policy, affected sources, target claims and correction generation
reject unsupported/stale/withdrawn decisions before effects
BEGIN bounded write transaction
    compare expected affected revisions and policy/correction generation again
    apply the declared candidate group through canonical record/lineage operations
    atomically persist resulting references + decision state + application receipt
    persist any required invalidation work / completed coverage for this unit
COMMIT
return receipt; separately establish read-view visibility

No model/network calls while holding the write transaction. Reuse canonical store logic inside a transaction-aware seam; do not call several independently committed upserts and describe the result as atomic.

The atomic unit is a candidate and its coupled support/supersession/invalidation obligations, or a small explicitly declared group of dependent candidates. Independent candidates may progress even if another is deferred. A partially applied batch must report its exact dispositions, not all-or-nothing fiction.

Compare-and-swap preconditions must cover the relevant claim neighborhood, including absence of a competing claim when creating one, not only rows the proposer already selected. Otherwise two concurrently validated contradictory additions can both become current. Use a scoped claim/source generation or equivalent existing mechanism; do not globally stale every batch for unrelated writes.

After commit/lost response, inspect the receipt before retrying. A stale lease is not proof a worker stopped: fencing/generation checks must prevent an old owner from committing after replacement. Exactly-once claims are limited to the proven scoped database effects; inference calls and consumer notifications can remain ambiguous.

Corrections, withdrawal and useful legacy data

Reuse #234 for currentness/withdrawal and #233 for lineage. Known corrections invalidate affected proposals and current read views immediately; they cannot wait for the next nightly dream to stop appearing as current. Protect paused/cancelled work from resurrection without restarting task machinery.

Retain original source statements separately from the current claim. Keep event time, ingestion time, asserted valid time, consolidation time and verification time distinct. Missing original time/author stays unknown. Reimporting old material is not fresh confirmation. Do not make an entire old corpus unusable because optional new metadata is missing.

Known derived links form a bounded dependency graph. Reject cycles; propagate invalidation and scope revocation through known derivatives, including stored proposals, cached packs/cards and pending presentation. Missing ancestry remains a documented uncertainty, not proof that every paraphrase was found. A claim supported only by revoked/private input must not leak through its summary; re-derive from a permitted subset as a new artifact if supported.

Guarded undo uses exact recorded effects and expected current revisions. It must preserve subsequent independent edits and known withdrawals. Do not restore a whole old database or blindly restore an archived claim to undo one batch. Backups/offline exports require their existing reconciliation rules; do not promise remote erasure of already-exported copies.

Prospective no-capture and retrospective erasure are distinct. An old queued proposal may contain private content even when accepted-memory rows do not; include proposal/validation caches in the actual privacy policy. Retain only permitted source spans and content-safe audit metadata, not an extra copy of every conversation.

5. Incremental execution and real economy

Keep one existing MemoryD scheduler/job owner. A generated batch, an applied batch and a source coverage watermark are different states.

  • Freeze a bounded source upper frontier. Use a stable ingestion/order cursor, or an equally precise source inventory, so late-imported old events and equal timestamps are not skipped. Original event time remains semantic metadata.
  • Distinguish “no cutoff supplied; choose default” from “explicit first scheduled scan with no previous cutoff.” Do not inherit the manual-apply watermark accidentally.
  • Advance generation coverage only over sources whose proposal/no-change disposition is durably recorded. Never advance to wall-clock now over an unprocessed capped tail. Adoption backlog uses its own durable candidate states; it must not block later independent evidence or disappear when generation advances.
  • Include bounded prior context/current claims for pronouns and contradiction checks. Mark retrieved context as context, not new evidence. Enforce a combined record/token/byte budget across source streams; existing records must not starve every newly captured turn.
  • Require changed evidence, due deferred work with new grounds, or an explicit bounded maintenance request before inference. A no-change tick performs zero model calls. Repeated reads/recalls are not new corroboration.
  • Use configurable batching, cooldown and idle eligibility, plus a maximum permitted age for low-volume pending evidence. Do not copy Honcho's thresholds or leave a user with two meaningful events waiting forever. A busy unrelated session should not starve closed evidence in another scope. A freshness target never overrides revoked permission, unavailable resources or a budget ceiling; report those specific backlog reasons.
  • Off the foreground path, run blocking provider work through a bounded existing blocking executor. Limit concurrency before enqueueing; dropping a task handle does not cancel its subprocess. Verify timeout/cancellation/process-group cleanup and foreground service responsiveness.
  • Respect the provider-off switch for all entry points. An automatic-policy revocation must prevent later in-flight adoption after revalidation; disabling learning does not itself erase existing memory.

Starting defaults for isolated acceptance, then measure

These are proposed initial policy values, not current settings or proven optimal thresholds:

one active consolidation batch globally in a small local deployment
one generator call + at most one validator call per batch; zero automatic retries
max 8 proposed candidates; request only the remaining available slots
per call: max 8,000 complete input tokens, 2,048 output tokens, finite deadline
max 32,000 bytes of selected evidence; enforce whole-wire/response limits too
idle target 5 minutes, cooldown 15 minutes, max pending eligible age 60 minutes

Account for exact tokenizer support versus estimates; reserve complete schema/system overhead and enforce reported usage afterward. A post-call rejection cannot undo provider spending, so reserve conservatively before dispatch and report overshoot honestly. Byte limits do not constitute tokenizer-exact guarantees.

The initial runtime policy must also have a finite rolling request/token allowance and, when applicable, a monetary ceiling. Persist reservations across restart, charge failures and ambiguous in-flight calls conservatively, and accumulate known command costs as well as HTTP costs. Subscription-backed inference has scarce quota even when marginal API dollars are zero. Unknown usage/cost is unknown, not free. Report requested, reserved, measured and unavailable telemetry separately.

For the implementation factory and consumer canaries, use currently available Luna-family models and GPT-5.3-Codex-Spark only, with explicit effective routing for proposer, validator, coordinator, reviewers and auxiliary calls. Luna low is the default; medium for nontrivial implementation; bounded higher Luna effort only for an evidenced blocker. Spark gets exact test commands, not permission to skip tests. No premium fallback or new API billing. Keep generic MemoryD provider support intact; this consumer model policy must not become an unnecessary core model monopoly.

Respect the existing task's total budget, two-worker aggregate cap and one-heavy-local-job cap; do not create new allowances per packet. A continuously configured learning service uses its separately approved rolling allowance, not a silently renewed overnight development allowance. No new scheduled job or consumer model-route change is activated just by opening this issue.

6. UX and recall acceptance

Expose a small CLI/status surface using existing service/protocol conventions. Suggested new vocabulary is consolidate run, consolidate status, consolidate inspect, and guarded consolidate undo; finalize once in the contract. Existing dream --preview and deterministic patch commands retain their documented meaning.

A useful status distinguishes:

  • configured policy/scope and effective learning mode;
  • provider configured, runnable, last successful inference and actual effective identity;
  • last generated batch versus last adopted batch;
  • committed changes versus read-view visibility/invalidation pending;
  • deferred/rejected/no-change counts and oldest actionable backlog;
  • effective limits, reserved/observed usage, error/backoff and next eligibility.

Status/inspection must not invoke a model, change memory, or claim that executable presence proves credentials work. “Saved” means an acknowledged durable write; “available to recall” requires the supported read path. Consumer prompt caches must not silently hide new memory until an unrelated daily reset.

The conversational experience should support these intents through existing native provider/tools where possible:

  • “Why do you remember that?” -> short source/perspective explanation, stated versus inferred, known time and current status.
  • “That changed” -> bounded supported correction, no second approval of the same explicit correction.
  • “Stop learning for now” -> actual scoped learning control, not a reassuring sentence.
  • “Undo that learning update” -> preview/execute only the supported, authorized inverse effects, with conflicts visible.

Automatically adopted material must be discoverable in ordinary task-appropriate recall. Preserve inferred labels, combine duplicate support, withhold invalidated current claims, and use bounded source drill-down rather than flooding the prompt. Reuse cards/pack invalidation instead of a new profile store. Consumer presentation should not announce every routine adoption or put a deferred queue on the user’s shoulders.

Fresh episodic continuity remains a separate fast path owned by #233; do not force every conversation to wait for Dreamer. Learned workflow descriptions remain memory. Turning them into executable skills/code needs the existing tested engineering/change process; Dreamer never rewrites identity/permission files.

7. Ownership and dependency boundaries

Existing owner Reuse here
#237 / #224 Typed inference, native auth, preview jobs and budgets; Packet A addresses relevant current regressions.
#231 Exact identity and source-supported legacy collision repair; do not invent another lossy deduplicator or repair missing originals with a model.
#233 / #228 Trustworthy producer/event identity, immediate episodic context and lineage. Implement the necessary narrow contracts together rather than duplicating an event store.
#234 Corrections, withdrawal, compatible legacy reconciliation and guarded current-state semantics.
#230 / #232 / #235 Native provider lifecycle, useful attributed task recall and acknowledged delivery/checkpoints.
#158 Procedure maintenance; no automatic procedure execution or second skill curator.
#193 Inert, scoped references; a known handle never grants access.

Do not make all of those epics blanket blockers. An unproven safety condition gates the affected automatic disposition; statement capture, persistence, tests and other independently safe work continue. End-state acceptance still requires both useful adoption and its relevant correction/recall guarantees—not a feature flag hiding permanently unfinished integration.

Consumer acceptance should be coordinated with the existing consumer learning/continuity owner and reviewed pinned-provider rollout. Keep its private operational receipts in its private workspace/repository. Do not copy private issue bodies, profile identifiers, real conversations, credential locations, financial data or live memory samples into this public issue or public fixtures. Source publication is not evidence of loaded-plugin activation.

8. Luna Factory execution packets

One coordinator owns integration and exact scope. Use the installed engineering skill and current repo instructions. Do not preload an entire skill library or hand every worker the entire repository/history. Each handoff includes: packet ID; pinned base; applicable contract version; exact owned files; expected input/output types; relevant fixture rows; tests; exclusions; current budget; expected artifact/receipt; and next owner. Preserve existing worktrees/writers.

Freeze the compact protocol and fixture manifest first. Packets are reviewable progress boundaries, not permission to stop the overall outcome after a plan or first PR. Do not create a second issue graph automatically.

A baseline/targeted repairs + B contract/fixture freeze
B -> C durable batches
B -> D evidence policy/validation
A + C + D -> E guarded application
E -> F existing scheduler/UX integration
G acceptance starts with B; final proof integrates A-F

Only disjoint C/D work should run concurrently after the contract is frozen. The same coordinator/integration writer owns shared src/store.rs, src/protocol.rs, src/service.rs and migration-number changes. Use small PRs or an isolated integration branch; no competing schema edits.

Packet A — baseline and necessary #237 repairs

Read/own: relevant paths in src/service.rs, src/provider.rs, src/status.rs, src/server.rs, src/config.rs, native companion, and existing command/config tests. Inspect actual scheduler entry-point location before changing it.

Input: pinned source + the eight grouped post-merge findings above; current ownership and baseline failure receipts.

Steps: write targeted failing tests; reproduce each material finding; fix the owning branch or consume an existing verified repair; rerun the exact tests. Keep authentication and preview-only defaults unchanged. Add no new provider or scheduler.

Output: exact repair SHA and supported preview/budget/provenance contract. Record baseline-only failures separately. A review discussion marked complete is not a passing test.

Tests: existing dream_command_scheduler, command_job_policy, command_no_http_fallback, config_env; add bounded single-Tokio-worker responsiveness and disabled-command/no-execution cases. Python companion suite must still pass.

Packet B — shared contract and fixed fixture baseline

Create: docs/dreamer-consolidation.md, tests/fixtures/consolidation/manifest.json, and compact protocol definitions in src/consolidation/types.rs with src/consolidation/mod.rs / module registration. These are proposed new files.

Input: Sections 3-6; current job/evidence/record/patch schema and #231/#233/#234 contracts.

Output: versioned ConsolidationPolicy, ConsolidationBatch, ConsolidationDecision, ConsolidationReceipt, source cursor/snapshot and stale-conflict definitions. Freeze operation/status enums and JSON examples before other workers code. Prefer references to existing domain types; no separate ontology taxonomy.

Tests: new tests/consolidation_contract.rs: closed-schema rejection, finite/size bounds, unknown optional legacy metadata, same-ID/different-output conflict, no request-side policy enablement and explicit empty cutoff. Commit fixed expected fixture dispositions before tuning.

Packet C — recoverable proposal storage and coverage

Own: minimal migration(s), transaction-aware store additions, narrow service preparation path, new tests/consolidation_store.rs; coordinate shared-file edits with the integration owner.

Input: Packet B types; existing dream_jobs/runs/evidence references; source/candidate identity contract.

Steps: reproduce lost returned-only proposal; persist immutable screened candidate payload and decisions; add scoped uniqueness/CAS; expose exact readback; implement finite source coverage with explicit cutoffs, equal-timestamp/late-arrival handling and unprocessed-tail continuation. Store in the canonical DB and preserve old handles.

Output: restart can load the same proposal/digest without rerunning inference; coverage never advances beyond durable work. Test disk-full/transaction failure, two creators, changed-payload replay, old-schema upgrade, and crash before/after proposal commit. Existing previews may retain their existing metadata audit; do not silently change them into adopting operations.

Packet D — evidence policy and bounded semantic validator

Own: src/consolidation/policy.rs, the smallest typed inference extension, new tests/consolidation_policy.rs, synthetic validator fixtures. Shared provider/protocol edits go through the integration owner.

Input: frozen batches, trusted source descriptors and current-claim/counterevidence snapshot. No database mutation authority in this module.

Steps: implement pure structural eligibility and unique-root counting; preserve source actor/time/perspective; add configured inference-only validation; map results into closed dispositions; compare near-duplicates without lossy equality; qualify inferred claims. Distinguish no support from provider failure and policy rejection.

Output: an inspectable decision with exact prerequisites, expected targets and bounded usage. No caller/model can manufacture user adoption, raise support by repeated references, or declare itself authoritative. Return defer with a reason where proof is incomplete; positive fixtures must still adopt.

Packet E — atomic adoption, correction and guarded undo

Own: canonical store/service transaction seam, new tests/consolidation_apply.rs, reuse existing correction/patch/lineage operations through their actual owner.

Input: persisted exact batch + decision digest + current configured policy; expected source/claim-neighborhood revisions and generation.

Steps: add apply transaction and durable receipt; atomically bind record effects/evidence/dispositions; make retry readback idempotent; reject concurrent correction/competing-claim races; invalidate current views; implement guarded inverse effects and suppression compatibility. Do not copy legacy permissive provider promotion or use conclusions as an escape hatch.

Output: one applied proposal yields durable, attributable current or inferred memory; lost acknowledgements do not duplicate effects; later changes survive undo. Raw model output never reaches upsert directly. Existing legacy HTTP automatic mode must either use the same governed policy boundary or be explicitly diagnosed/gated until migrated—not silently bypass it.

Packet F — scheduler, limits and user-facing read/control path

Own: existing scheduler entry point, src/cli.rs, src/status.rs, src/config.rs, src/protocol.rs, relevant native adapter rendering, docs; new tests/consolidation_scheduler.rs and tests/consolidation_cli.rs.

Input: A/C/D/E contracts; finite policy budget and operator-selected mode/scope.

Steps: wire the new end-to-end automatic operation behind its explicit mode; enforce one bounded owner and rolling reservations; implement zero-model no-change behavior and pending-age fairness; keep inference off foreground service threads; report generation/adoption/visibility separately; expose inspect/correction/pause/undo through supported permissions. Test preview/disabled compatibility and loaded versus configured state.

Output: unattended eligible evidence progresses to usable memory without per-candidate approval, while the user can inspect and correct it. No consumer reset, extra daemon, automatic private archive sweep, or premium route.

Packet G — end-to-end and existing-corpus acceptance

Start with B; finish after A-F. Own: new tests/consolidation_journey.rs, fixed fixtures and docs/dreamer-consolidation-acceptance.md; reuse existing native provider integration harness.

Input: identical sanitized old/new stores, frozen prompts/expected properties, the same total recall/inference budgets, exact runtime/model/provider versions.

Output: compare preview-only baseline, deterministic-only baseline and governed candidate on the fixed workload. Show durable adoption, better subsequent answers and correction survival in a fresh real consumer, not only store helpers. Report actual sample counts, failures, costs/unknowns, component versus end-to-end latency, configuration and rollback needs.

Use temporary DBs/disposable profiles and owned loopback processes. Existing authorized auth stays with its owner; never copy credentials into fixtures/logs. Unavailable live-model evidence remains NOT VERIFIED, not permission to substitute a premium model or mark acceptance complete. Keep deterministic tests shippable while that gate is unresolved.

Minimal executable assertions for the new suite

Illustrative test-contract pseudocode, not a claim these helpers already exist:

given automatic policy and a captured ordinary user preference
when the actual consolidation service completes and a fresh consumer recalls
then one attributed memory is committed, useful context is returned,
     and no per-memory approval was requested

given a stored proposal citing a real but irrelevant source
when semantic support is checked
then no unsupported statement is adopted despite the valid reference

given a validated proposal and a later explicit correction
when the old proposal applies
then stale-precondition conflict, zero old-claim resurrection

given an adopted batch, a lost reply and process restart
when the same batch/decision is retried
then the same durable receipt/effects, no new evidence weight

given two opposite claims validated against an empty neighborhood
when two workers commit concurrently
then at most one becomes current; the other revalidates/defer-conflicts

9. Mandatory fixture matrix

Freeze fixtures first. Mix positive, negative, temporal, recovery and small-corpus cases. Use synthetic ordinary names/projects; no real user archive is needed for committed tests.

ID Input / perturbation Required result
P01 User plainly states a non-sensitive stable preference once. Automatically adopted as an attributed statement and used in a fresh session, with no approval prompt.
P02 Same words inside a quotation, hypothetical example, joke, negation or “only for this task.” No unqualified lasting preference; preserve the actual scope/uncertainty.
P03 Two independently captured relevant events support a useful non-sensitive pattern. An inferred belief is admitted and recalled as inferred, not “the user explicitly said.”
P04 Same event copied through two clients, repeated evidence IDs, summary re-ingestion. No new primary roots/confidence or duplicate current belief.
P05 Valid source ID whose content does not support the claim; compound claim with one unsupported clause. Unsupported portion never adopted; revise into a new reviewed digest or defer/reject.
P06 Current explicit correction followed by old import, delayed dream and restored old cached pack. Correction governs current recall; stale outputs cannot re-adopt the known wrong claim.
P07 Repeated paraphrases over 20 synthetic cycles with unchanged primary evidence. Bounded candidate/history retention and no growth in equivalent active claims or support; no-change cycles use no inference.
P08 git branch -d/-D, /srv/Alpha/alpha, indentation, units, quantities, language and negation differences. Preserve distinct meanings; similarity does not merge them.
P09 Same display name in different scopes; known foreign handle; revoked source grant. No cross-scope access or derived-content leakage; identity is not inferred from names.
P10 Assistant-authored conclusion with target=user; worker reports success but actual test fails. Neither becomes a user assertion or verified successful outcome.
P11 Two model outputs for one input; exact replay of one output. Distinct proposal identities; exact replay returns identical receipt without duplicate effects.
P12 Crash before proposal persistence, after persistence, during apply, and after commit before response. No acknowledged-but-lost proposal; recover durable progress; atomic effects; inspect-before-retry.
P13 Concurrent correction, two opposing additions, policy revocation and stale owner after lease expiry. Revision/generation fence prevents stale or unauthorized commit.
P14 Manual apply watermark exists, no scheduled watermark; first scheduled batch is capped. Explicit no-cutoff preserved; all eligible tails remain discoverable and are processed without starvation.
P15 Equal timestamps, backdated import, clock skew, missing source date, multiple source streams. No skipped ingestion; honest time basis; one combined budget; no invented recency.
P16 Low-volume evidence, continuously busy unrelated scope, unchanged idle tick. Useful evidence eventually progresses within declared freshness policy; no model-based polling or required 50-message threshold.
P17 Slow command on single Tokio worker; shutdown or cancellation while blocked. Foreground recall remains responsive; bounded owned process cleanup; no late adoption from cancelled generation.
P18 Disabled command, spoofed provider identity, repeated daily command costs, full-prompt overrun, exhausted quota. Actual off/budget boundary enforced with truthful provenance, min effective limits and no premium fallback.
P19 Source/metadata contains prompt injection, secret-shaped data or hidden-reasoning payload. Screen before proposal persistence; no secret echo or authority change through quoted text.
P20 No-capture interval or known withdrawal reaches a queued proposal and its derivatives. Follow the real retention/suppression policy across proposals and current views; no delayed privacy leak.
P21 Undo one batch after another legitimate edit/correction; restore a pre-withdrawal backup. Preserve later work; conflict/remaining reconciliation explicit; no whole-DB rollback masquerading as safe undo.
P22 Mixed pre-change corpus with safe useful old records, unknown time/lineage, transferred origins and prior corrections. Existing handles/meaning preserved; bounded useful legacy recall remains; no automatic corpus re-key or fresh-authority laundering.
P23 One candidate needs judgment, others are straightforward; validator unavailable for another. Safe independent candidates progress; specific deferral does not freeze the batch or cause a queue of user approval prompts.
P24 Consumer process/session reset and swap to another available allowed model; built-in memory/transcript controls. Same permitted durable knowledge/corrections retrieved through real interfaces, with no answer leaked in the new prompt.
P25 New adopted memory exists but adapter cache/view remains old; source invalidation during recall. Saved versus usable is distinguished; actual next fresh read reflects current disposition within declared consistency semantics.
P26 User rejects an inferred insight; proposer returns a paraphrase with no new evidence. Rejection respected, no repeated interruption or new supposedly independent memory.

Add property/perturbation tests for irrelevant-event reordering, duplicate callbacks and tiny budgets. Where a dependency is missing, report the exact unsupported behavior; do not delete the failing case or replace it with a mock that assumes the desired result.

10. Verification commands and completion gate

Run the relevant existing suites on the pinned base first, then the candidate. Use existing pinned/RCC/container environments where appropriate. No unbounded local compile fan-out.

cargo fmt --all --check
cargo check --locked
cargo test --locked --lib
cargo test --locked --test dream_command_scheduler --test command_job_policy --test command_no_http_fallback --test config_env
cargo test --locked --test dream_jobs --test dreamer --test dreaming_fixtures
python -m unittest discover -s tests -p test_dreamer_native_provider.py -v

The new test targets to create in Packets B-G are:

cargo test --locked --test consolidation_contract --test consolidation_store --test consolidation_policy
cargo test --locked --test consolidation_apply --test consolidation_scheduler --test consolidation_cli --test consolidation_journey

Also run impacted existing migration, temporal-recall, native-adapter, policy, patch/undo and contract-snapshot tests after verifying their actual target names. Baseline failures do not become waived acceptance: identify their source, fix relevant regressions, and distinguish untouched known failures from a green full suite. Do not rewrite expected answers to fit the candidate.

Definition of done

  • Existing deterministic/typed preview contracts remain valid; new automatic mode is explicit and actually applies the exact model proposal through governed policy.
  • Positive statement and inferred-belief fixtures are automatically adopted and useful without per-memory approval. Always-defer/always-reject fails.
  • Durable candidate/decision/receipt state survives restart and retry with tested atomic effects and bounded storage.
  • Corrections, source revocation, known withdrawals and guarded undo work across proposals, records and current consumer views.
  • Scheduler coverage does not skip capped/late/first-run evidence; background inference cannot monopolize foreground service or bypass off/budget controls.
  • Existing corpus/identities/source times remain useful and honestly represented; no clean-empty-store-only acceptance.
  • Fixed-workload report includes useful recall/adoption coverage, unsupported adoption, duplicate-current-claim growth, missed corrections, unnecessary questions, leakage, request/token costs, and p50/p95 foreground/background timing with sample counts.
  • All prohibited effects in the fixture matrix have zero occurrences; positive recall is nonzero and meets frozen fixture expectations. Passing a finite suite is not a universal reliability claim.
  • A real affordable-model generation -> governed adoption -> fresh native consumer recall -> correction -> restart journey is demonstrated separately from deterministic mocks. A second-model portability gate is labelled accurately if unavailable.
  • Exact-head independent review is resolved, including current feat: support subscription-backed Dreamer previews through native providers #237 findings relevant to automatic learning; actual test receipts are inspected, not just worker claims.
  • Documentation includes mode/scope onboarding, effective status, preview compatibility, source explanation, correction/pause, budget control, compatible upgrade and guarded rollback. Consumer deployment uses reviewed pins and preserves user-owned state.
  • One bounded production activation decision and its rollback prerequisites are ready for the owner. No live activation or private-corpus migration is claimed from a merged PR alone.

Final handoff: exact commits/PRs; generated versus adopted versus consumer-visible evidence; effective models and measured/reserved usage; supported legacy/mode matrix; genuine remaining gates; and the next authorized step. Keep evidence concise and public fixtures synthetic.

The milestone ends with quietly useful, inspectable, correctable automatic memory, not a larger preview log, a new approval dashboard, or an ontology/factory rewrite.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions