diff --git a/dsh-mneme/CHANGELOG.md b/dsh-mneme/CHANGELOG.md index cc360d2e..09e109dd 100644 --- a/dsh-mneme/CHANGELOG.md +++ b/dsh-mneme/CHANGELOG.md @@ -2,6 +2,14 @@ ## [Unreleased] +- **跨类型同义双录：蒸馏提示词里「宁可拆成多条」把同一件事记了两遍，而三处去重各守一道类型闸（#687）；实体表 75% 的实体只被提过一次、名字只按大小写敏感精确匹配（#688）**。两条 issue 报告的是同一个根病：**记忆的「类型」被当成了「对象身份」**。#687 上报者（xueyue33）给的分布很刺眼——他自己库里 `sim>=0.85` 的 81 对近重复中 **72 对跨类型**、`>=0.90` 的 7 对中 6 对跨类型；`sim>=0.88` 的 24 对里 21 对跨类型。四处夹缝逐个核过都成立：`src/lang.js` 的 `codingSummary` 明确让模型「宁可拆成多条」而 `pitfall`（现象+根因+规避）与 `constraint`（约束+来源）本就是同一结论的两种写法；`src/service.js:1480` 的 `findSessionDuplicate` 用 `store.list({ type: memory.type, source, limit: 200 })` 锁死同类型同会话；`src/dream/clustering.js:117` 的 `findPotentialConflicts` 首行 `if (memories[i].type !== memories[j].type) continue;`；dream 巩固提示词同样写着「仅合并同类型」。**我们不开常开类型闸**：本机活跃库 9,275 条带向量的记忆里跨类型 `cosine>=0.90` 共 45 对，逐对读正文后 **36 对（80%）确为同义，9 对合并会丢信息**（5 对时间序列/状态快照、2 对症状 vs 修复、1 对结论反转、1 对版本号不同）——也就是说「凡是跨类型高相似就合并」每 5 对要吞掉 1 对真互补的信息，而这类对只占活跃总量 0.97%，收益与风险不对称。落地的三层是**源头互斥 + 批内去重 + 存量一次性处置**：① `codingSummary` 提示词（中英双语）补一条互斥约束——同一结论只记一条，`pitfall` 与 `constraint` 不双写，时间序列/状态快照写进同一条；② 蒸馏落盘前做**批内**跨类型去重，只对 `pitfall↔constraint`、`history↔project` 两组生效（正是实测最集中的应用对：9 对 + 8 对），`planBatchCrossTypeDedupe(entries, { minSim })` 是纯函数——按相似度降序、每条最多被退休一次、不许链式藏行（A 取代 B 的同时 B 又取代过 C 会把 C 永久埋在链底）；门槛 `BATCH_CROSS_TYPE_MIN_SIM = 0.85`（textSim，不是余弦，词面重合是这两组对的主特征）、`BATCH_CROSS_TYPE_MIN_CHARS = 24`、`BATCH_CROSS_TYPE_MIN_LEN_RATIO = 0.5`；③ 处置走 `store.supersedeChain` 只写 `superseded_by` / `superseded_at`，**保留正文更全的一条、被取代者正文零改写、不归档、不 merge**——判错时旧行仍在库里、显式搜索照旧读得到，且 `supersededIds()` 的 `JOIN ... AND w.archived=0` 让继任者一旦被归档旧行自动回到可见，等于自带回退。刻意**没有**复用 `src/dream/decisions.js` 的 `supersedeInfoFirstPair`：它走 `service.update` 会在正文尾部追加「已被取代」前缀，与「正文零改写」直接冲突。**默认开**（2026-11-13 拍板）：这个开关存在的意义就是让新写入不再双录，默认关等于修了不生效，所以 `src/config.js` 给 `true`、设 `false` 是 kill switch，且关时审计行的 `batchDeduped` / `batchPairs` 两个键根本不出现——加开关之前的行为逐字节不变。两个键同时登记进 `settings.js` 的 `FEATURE_FLAG_BOOLEANS` 与 `FEATURE_FLAG_NUMBER_RANGES`，面板上能改。存量 17 对**不由代码处置**：只读脚本 `scripts/cross-type-dedupe-dryrun.mjs` 出清单（输出**禁止落在仓库内**，清单里含记忆原文），复核后由人决定、按纯 SQL 打链。 + + **#688 的修法只落在写入侧**。上报者库里 12,700+ 实体、38,500+ 关系、孤儿 41%、`mention_count=1` 占 72%；本机 live 库（只读打开）实测实体 **25,474**、关系 **59,715**、`mention_count=1` 占 **75%**、大小写/空白重复 262 组；而按 `entity_relations.memory_id` 真实连接口径，孤儿是 **17.9%（4,560 行）** 而不是 41%——口径不同，这里写清楚。存量**不做合并**：实体 id 是 `randomUUID()`，合并要重指 59,715 行关系且 `entity_relations` 没有唯一约束，重指过程一旦中断会造出重复边，收益（0.68% 的实体）与风险同样不对称。改的是查找键——`findEntityByName` 从 `WHERE name = ?` 改成 `lower(trim(name))` + `type` 的等价比较并为此建表达式索引（`DeepSeek-V3` 与 `DeepSeek-V4` 是两个不同对象，不剥版本号后缀；`createEntity` 落库只去首尾空白、**保留原大小写**用于展示）。类型维度不能省：实测去掉 type 会误并 95 组（`constraint` 与 `pitfall` 常同名）。另外 `entities.canonical_memory_id` 是 v0.7.1（`5b5a2aa4`）引入的预留列，**从没接通线**——`src/store.js` 里只有 DDL、行映射、`INSERT`、`UPDATE` 四处 plumbing，没有任何调用方写它或读它，25,474 行全 NULL；这次按幂等迁移删列（`PRAGMA table_info` 判断 + `ALTER TABLE ... DROP COLUMN`，宿主 SQLite 3.51.3 支持）。迁移成本实测：25,474 行旧 schema 表首次 `createStore` **396ms**（含建表达式索引），二次 24ms。 + +**升级不可逆，降级前必须备份**：`ALTER TABLE ... DROP COLUMN` 是单向门——旧代码里没有 `addColumn`，把插件降回仍带 `canonical_memory_id` 的版本后，`UPDATE entities ... canonical_memory_id` 会报 `no such column`；`src/entities/extractor.js` 逐实体 catch 后 `extractEntities` 降级为 `{ ok: false }`，即**记忆写入照常、实体与关系停止落库**（warn 变多），直到重新升级。历史库本身安全：`sqlite_master` 里没有引用该列的 view / trigger / index。 + + **测试** 3,488 → **3,496**（+8）：`test/summarize.test.js` 覆盖默认路径打链、kill switch 完全不碰、跨类型对之外的组不并、链式藏行被拒、`dup` 已有值跳过；`test/entities.test.js` 覆盖大小写/空白命中、版本号不归一、type 隔离、旧库补列删列幂等。变异自证：把默认改回 `false` → 2 红；`MIN_SIM` 降到 0.5 → 2 红；去掉 type 维度 → 1 红；`lower(trim(...))` 改回精确匹配 → 2 红。另外如实记一处**两份尺子**：dry-run 那 17 对是按 `cosine>=0.90` 挑的，线上规则判的是 `textSim>=0.85`，报告里 `textSimBands` 是 `0.9-1:1 / 0.8-0.9:2 / 0.7-0.8:3 / 0.6-0.7:5 / 0.5-0.6:6`——这 17 对里只有约 1–3 对会被线上规则真认出来，所以存量清单**不能**当作「新规则会抓到什么」的证据，线上规则偏保守是设计。 + - **仓库地志的身份与位置分家：`portable_key` + 每机位置表，双系统 / 双机共享同一个 memory.db 时地志第一次可以跨端复用（#581）**：mneme 把「库共享」做到了（`memoryDir` + WAL + busy_timeout），chorography 是唯一一处因为记了本机绝对路径而破坏这个前提的功能——Windows 端登记的行，macOS 端拼出来的路径不存在；再跑一次 T0 又多出第二行，从此不带 `repo` 参数的调用命中 repo-ambiguous；T1 的逐字符严格匹配更意味着**每端都必须先各自跑一次 T0**。修法按 issue 的最小方案落地，三个概念各就各位：**身份** = `choro_repos.portable_key`（新列，幂等迁移），取 `remote.origin.url` 归一化——协议头 / 尾斜杠 / 尾 `.git` / 大小写剥平，scp 形态（`git@host:path`）的冒号换 `/`，**userinfo 整体剥掉**（token 不进键：泄进库里是一回事，令牌轮换后同一仓库算出不同键是另一回事）；无 remote 退化到**根提交哈希**（`git:`，跨副本恒定且与「碰巧同名的别的仓库」天然区分——issue 提的「退化为仓库名」不能用，目录名与 package name 重合率太高，第二次登记会吃掉别人的行，实现期在测试夹具上当场撞到）。**位置** = 新表 `choro_repo_paths(repo_key, host_id, root_path, last_seen_at)`，`host_id = os.hostname()`，T0/T1 登记时 upsert。**读侧**（`queries.js` 的 `localRootPath`）按当前 host 取本机路径、取不到回退身份行的 root_path——单机库两个值恒等、行为逐字节不变；本机没登记过时落进既有的 page-missing / unreadable 失败口径，不会拿别的机器的路径读出错误内容。**T1 的登记校验**从「root_path 逐字符相同」改成「本机路径命中**或**身份已登记」——同一仓库在另一台机器登记过，本机直接认领那一行富化，消灭「每端先各跑一次 T0」；认领同时落本机位置行。写路径（`/choro/run`、`/choro/t1` 后台任务、`/choro` 命令的过期判定）同样改吃本机根。**两个刻意的取舍**：① 同键登记收敛到同一行，但 root_path 的 UPDATE 前先查目标值是否被别的行占着（被占保留原值、位置走位置表）——升格期已跨端登记出的重复行**不合并**（页面与 claims 挂在 repo_id 上，机器间合并行等于改写归属），强行改会撞 `idx_choro_repos_root` 直接抛；② 升格存量行的键留 NULL，由它自己的机器重跑 T0/T1 时补上，不猜。已知边界：kb 导入的 choro_page source 是导入当时的绝对路径快照，跨端复用时会失配，另账处理。回归测试 +10（URL 归一化形态表含凭据剥离 / 无 remote 退化链与恒定性 / 旧库补列补表且重开幂等 / 位置表同键同机覆写 / 同键跨端登记收敛同一行且 created_at 不动 / 升格期 root 被占不强改 / 读侧三 host 各解析各的 / **T1 跨端认领端到端**：克隆检出直接富化身份行并登记本机位置），另把四个「同库登记两个相同夹具」的既有用例显式区分开（无 remote 键是根提交哈希，同秒提交的相同内容会同键收敛——这本身就是退化链设计的防误并性质）。变异自证 3 例（删 `upsertChoroRepoPath`/`choroRepoPathFor` 的 return 出口 → store 用例 3 红；读侧 `localRootPath` 去掉位置行解析 → queries 用例红；T1 去掉 byKey 认领 → 端到端用例红），恢复后全绿。**评审补丁（PR #679，六条 Major 全部成立）**：① **路径回退要核验身份**——两台主机的同一条绝对路径可能各自检出不同仓库，`upsertChoroRepo` / T1 的登记校验、读侧 `localRootPath` 都改成「身份优先，路径回退只接受无键存量行或键一致的行」；核验用 `portableKeyFor`（一次 git 读取，按「键+路径」记忆化），不符时读侧返回空串并按 `page-missing` / 不可读上报，绝不 `join("", rel)` 去读进程 CWD；`claimDrift` 的 `hasT1` 同样加了空根守卫。② **身份唯一**：`portable_key` 加部分唯一索引（`WHERE portable_key IS NOT NULL`），插入撞唯一键时回读同一身份行改走更新——并发登记不再产生重复行；索引建失败（存量重复键）退成非唯一索引，不让插件起不来。③ **URL 解析分清端口与 scp 分隔符**：带协议头时 authority 的冒号是端口（默认端口 ssh 22 / https 443 / git 9418 剥掉，其余保留），只有 scp 形态的冒号才是路径分隔符；④ **userinfo 只在 authority 段内剥**，仓库路径里的 `@`（`repo@stable`）不再把主机与路径一起切掉。⑤ `repoByCwd` 的最长前缀改为比较候选自身路径长度（原实现拿身份行里那条外地路径比长度，内层本机路径永远赢不了）。新增回归 5 条（端口 vs 路径段 / 路径里的 @ / 同路径另一身份拒绝且原行不变 / 回退目录身份不符报 page-missing / 前缀比较用候选长度），变异自证 6 例逐条变红。**二轮评审补丁**：① **写路径补空根守卫**——`/choro/run` 原先直接吃 `localRootForRepo` 的返回值，该方法在本机未登记位置、且回退目录身份核验不过时返回空串，`resolve("")` 即宿主 cwd，T0 会把那里的模块与页写进这条 repo 行（「同一路径不同仓库」事故的另一半：读路径挡住了、写路径漏了）；现在空根直接 409 `repo-not-reachable-here`，并对旧装配沿用 `typeof` 守卫退回 `root_path`。② **userinfo 只在主机段内剥**：scp 形态先在冒号处切开再找 `@`，`github.com:team/repo@stable` 不再退化成整段丢失的 `stable`。③ 唯一索引建失败退普通索引时补一行 warn，不再静默降级。新增回归 2 条（空根 409 且不触 T0 / 旧装配缺方法退回 `root_path`），URL 形态表补 5 条断言；变异自证 4 红（撤守卫 + 退回全串 `indexOf("@")`）。 - **取代链与遗忘的接缝修好了：继任者被 `memory_forget` 之后，被取代的旧值不再回潮成为「当前答案」（#532 存储层半边）**：#532 主楼的现象是「更正两个事实、只忘一个，保留的那个答回更正前的旧值」。`docs/FORGET.md` 的定案把根因拆成三段接缝（更正落新行且旧行活跃 / `memory_forget` 行粒度 / 旧值行成了唯一幸存载体），其中第二、三段的交互有一个**确定性**缺陷：`#650` 给 `supersededIds()` 加的守卫把「继任者归档、遗忘、删除」统一当成「继任者退场」，于是继任者一旦被遗忘，被它取代的旧行就重新回到注入面——而 `memory_forget` 的语义恰恰是「我不要再记得这个内容」，让用户明确抛弃的旧答案借继任者的退场回潮，正是本报的现象。现在两种情况分开：**遗忘继任者**不再恢复旧行可见性（旧行仍留在 suppressed 集合里、注入面不选它，但显式搜索照旧读得到——`fuseRecall` 只降权 0.5，不是沉默的不可逆失效）；**归档继任者**维持 #650 原语义恢复可见（归档是整理面退场、可恢复、且常是合并落点，旧行可能是唯一载体），并加回归锁防止这条修正顺手改掉它。`memory_forget` 的回执同时报出连带面：新增只读 `store.supersedeContext(id)`（`{ supersededBy, predecessors }`）经 service 透传，抑制一行时若它取代过 N 条更早的记忆，回执明写「它们保持隐藏（显式搜索仍可达）、不应被当成当前值」——模型不必自己猜；恢复与无链行不渲染那句话。**明确没修的部分**（写进 `docs/FORGET.md` §5，别读成闭环）：没有取代链的库（dream/sleep 未运行或 `conflictLedgerMode` 关着）里 `memory_forget` 依旧行粒度，§4 合成轨迹的「旧值幸存」断言依旧成立——写侧自动识别「同主题改值」需要 LLM 判断进热路径，仍按 R2/R3 等排期。回归测试 7 条（遗忘继任者旧行保持抑制且不入注入面 / 显式搜索仍可达 / 恢复继任者后仍抑制 / 归档继任者按 #650 恢复 / 回执报连带面且恢复时不报 / `supersedeContext` 空行返回 null 与计数只认指向本行的行 / **旧装配缺 `supersedeContext` 时回执降级、遗忘本身照常完成**）。最后一条补齐了审计发现的一处健壮性缺口：工具层原本裸调 `service.supersedeContext`，旧装配或手写测试替身没有该方法时会抛 `TypeError` 把整条遗忘操作打掉——回执增强不该有否决权，现按「无链信息」降级（与 `service.js` 的 store 层守卫对称）。变异自证 5 例（把 `forgotten = 0` 条件加回去 → 1 红；去掉 `archived = 0` 守卫 → 1 红；回执 `predecessors` 恒 0 → 2 红；恢复时也渲染连带面 → 1 红；删掉工具层 `typeof` 守卫 → 1 红），恢复后全绿。**评审补丁（PR #681，两条 Minor）**：回执只承诺「现在真的成立」的事——`supersedeContext` 增 `chainActive`（与 `supersededIds` 同判据：继任者被归档 ⇒ 旧行已恢复可见，不再声称被压住）与 `activePredecessors`（仍活跃的前序条数）；渲染句改成「不计为当前值」，可查承诺只给活跃的那部分（`the N that remain active`），全退场时不作可查承诺。变异自证 3 例（归档后仍称链生效 / 可查计数不筛活跃 / 渲染不看 `chainActive`）逐条变红。 - **设置页加搜索：一张框扫全部组，两层折叠里的行也够得到（#653）**：设置页攒到 30+ 个开关后，「新增功能找不到入口」成了反复出现的抱怨（#607 的知识库、#653 的原话「浏览器的搜索功能在设置页面也搜不到『知识库』」）。浏览器 Ctrl+F 搜不到的机理很具体：**收起的组根本不在 DOM 里**——「全部设置」与「高级」两层折叠把不渲染的行藏在了 React 树之外，页内查找无从谈起。修法是在设置页顶部加一个常驻搜索框：有词时目录（#518）让位（跳转目标多半已被过滤掉，留着只会点到没反应），五组卡片按节过滤——**组标题或该组任一卡标题命中则整节原样显示**（搜「连接」应该看到全节而不是几行），**功能开关卡则切成命中行平铺**，两层折叠被绕开：折叠里的行恰恰是搜索最该够到的地方。行目录从六张常量表拼出来（`FEATURE_GROUPS` / `FEATURE_CHILDREN` / `FEATURE_ADVANCED_BOOLS` / `FEATURE_ENUMS` / `FEATURE_STRINGS` / `FEATURE_NUMBERS`），刻意不立第二份清单——任何一张表被绕开，那一层的行就从搜索里消失；四个没有通用渲染器的枚举键（embedProvider / 思考强度×2 / 地志档位）经 `SEARCH_ENUM_SOURCES` 补登记，取值直接引用各自已有常量。搜索分支的行渲染器与正常视图**同一份**（`boolRow`/`enumRow`/`strRow`/`numRow`，`enumRow` 为此加了 `valuesOverride` 形参而不是被克隆一份）——搜索路径改的行与正常路径逐字一致。子行在搜索结果里不缩进、不带 gate 标注：搜索结果是「这个键存在、可在这里改」，父子关系留给正常视图讲。全搜无果时有明确空态（点名搜索范围是名称/说明/键名），不是白屏也不是空卡。无新配置键（搜索不改任何行为，是纯视图过滤）、无新 CSS 色声明（布局走内联样式与既有类，#178 的对比度锁不受影响）。回归测试 9 条：索引必须走全六张表、**索引里每个键在双语词典各有一条标签**（76 键全量核对，缺一侧就会有一侧把键名当标签渲染）、目录让位与空态在场、`FeaturesCard` 唯一调用点且搜索分支复用四类渲染器、四条 i18n 键恰好中英各一条，以及评审补的四条：**每个用字面量渲染的行键都必须登记进某张可搜表**（`injectRotationTurns` 此前漏登记——它挂在自己的总闸下、不在任何组项数组里，只断言「索引引用了六张表」抓不到这一类）、搜索结果分支必须渲染 `saveError`（PUT 失败时控件回滚，不显示就成了「改了没反应」）、组内每张卡片标题都要在 `SECTION_TITLE_KEYS` 里（`configKeys.title` 此前缺席）、搜索框与空态文案必须分清「开关按行匹配」与「分组/卡片按标题匹配」。变异自证 8 例（索引删 `FEATURE_ADVANCED_BOOLS` / 删 `FEATURE_ENUMS` / 目录不让位 / enumRow 不传 valuesOverride / 漏登记 `injectRotationTurns` / 搜索分支不渲染 saveError / 漏登记 `configKeys.title` / 文案回退 → 各自变红），恢复后全绿。 diff --git a/dsh-mneme/docs/CONFIGURATION.md b/dsh-mneme/docs/CONFIGURATION.md index d47c1129..af2aea66 100644 --- a/dsh-mneme/docs/CONFIGURATION.md +++ b/dsh-mneme/docs/CONFIGURATION.md @@ -136,6 +136,8 @@ | `summarizeDedupeMode` | `off` | 落库前去重档位：`off` / `title`（拦完全同名）/ `vector`（同会话语义近邻） | `vector` 复用已有 embedding 列，无 LLM 调用 | | `summarizeDedupeMinSim` | `0.92` | `vector` 档并入阈值（0.5–0.99） | 仅 `vector` 档生效 | | `summarizeDedupeWindowHours` | `24` | `vector` 档回看窗口（小时，≤168） | 仅 `vector` 档生效 | +| `summarizeBatchCrossTypeDedupe` | `true` | 批内跨类型去重（#687）：同一窗蒸馏产出里，把重复的 `pitfall`/`constraint` 或 `history`/`project` 两组类型对合成一条 | **默认开**（2026-11-13 拍板；设 `false` 即关，行为与加这个开关之前逐字节一致）；只认这两组类型对（其余类型对实测有 20% 会丢信息）。处置是 `supersede`：被取代的一条**正文零改写**、不归档，只是不再默认注入——判错也不丢内容 | +| `summarizeBatchCrossTypeMinSim` | `0.85` | 批内跨类型去重的相似度阈（0.5–0.99） | 仅上一项开启时生效；与质量过滤的 duplicate 信号同档，避免同一条「是不是同一件事」出现第二个数 | | `distillRateLimitIntervalMs` | `1000` | 蒸馏请求全局串行排队的相邻间隔（毫秒） | 0 = 关闭排队（默认开的 429 保护随之失效） | | `distillRateLimitRetries` | `3` | 命中 429 的指数退避重试次数（≤10） | 重试全程对用户透明 | | `distillRateLimitBaseDelayMs` | `1000` | 429 退避基数（毫秒，100–60000） | 1s→2s→4s… | diff --git a/dsh-mneme/docs/ENTITIES.md b/dsh-mneme/docs/ENTITIES.md index 4235ba09..786dd170 100644 --- a/dsh-mneme/docs/ENTITIES.md +++ b/dsh-mneme/docs/ENTITIES.md @@ -28,11 +28,15 @@ CREATE TABLE IF NOT EXISTS entities ( type TEXT, -- person|project|concept|technology|organization first_seen TEXT NOT NULL, -- 首次出现时间 last_seen TEXT NOT NULL, -- 最近出现时间 - mention_count INTEGER DEFAULT 1, -- 提及次数（同名去重后递增） - canonical_memory_id TEXT -- 规范来源记忆（保留位） + mention_count INTEGER DEFAULT 1 -- 提及次数（同名去重后递增） ); CREATE INDEX IF NOT EXISTS idx_entities_name ON entities(name); CREATE INDEX IF NOT EXISTS idx_entities_type ON entities(type); +-- #688 写入侧归一：命中键在 SQL 里折叠（lower + trim），索引与查询同源。 +-- 必须带 type 维度——只按 name 折叠会把 95 组跨类型同名实体误并成一个 +-- （如 DeepSeek 这个技术 与 deepseek 这个组织）；也不剥版本号后缀， +-- DeepSeek-V3 与 V4 是两个对象。存量行不改写，只在写入/查找时归一。 +CREATE INDEX IF NOT EXISTS idx_entities_name_key ON entities(lower(trim(name)), type); ``` ### `entity_attrs` — 属性（快照式时间轴） diff --git a/dsh-mneme/lib/client.js b/dsh-mneme/lib/client.js index 51383b12..4a8b396f 100644 --- a/dsh-mneme/lib/client.js +++ b/dsh-mneme/lib/client.js @@ -1045,6 +1045,10 @@ window.__ModuleLoader__.load({ "memory.features.codingBoostFactor": "编码记忆加权", "memory.features.memoryQualityFilter.enabled": "质量过滤", "memory.features.memoryQualityFilter.enabled.hint": "低价值记忆自动归档、注入时降权。", + // #687：写侧去重，与质量过滤相邻——两条都在问「这一条该不该成为一条记忆」。 + // 文案必须说清「原文不删」：被取代方只是不再默认注入，用户不会丢内容。 + "memory.features.summarizeBatchCrossTypeDedupe": "批内跨类型去重", + "memory.features.summarizeBatchCrossTypeDedupe.hint": "同一轮蒸馏里把重复的「踩坑/约束」或「历史/项目」只留信息更全的一条；被取代的一条原文保留、不再默认注入。", "memory.features.llmAudit.enabled": "调用审计", "memory.features.llmAudit.enabled.hint": "后台模型调用与 token 消耗都有账可查。", // #254：写入准入两层。检测关时整条路径不碰；检测开、拦截关 = 仅告警 + @@ -2192,6 +2196,10 @@ window.__ModuleLoader__.load({ "memory.features.codingBoostFactor": "Coding boost factor", "memory.features.memoryQualityFilter.enabled": "Quality filter", "memory.features.memoryQualityFilter.enabled.hint": "Low-value memories get archived and down-ranked.", + // #687 — write-side dedupe. The hint must say the loser keeps its text: + // supersede only stops default injection, it does not delete content. + "memory.features.summarizeBatchCrossTypeDedupe": "Batch cross-type dedupe", + "memory.features.summarizeBatchCrossTypeDedupe.hint": "Within one distillation run, keeps only the fuller of a duplicated pitfall/constraint or history/project pair; the superseded row keeps its text but stops being injected by default.", "memory.features.llmAudit.enabled": "Call audit", "memory.features.llmAudit.enabled.hint": "Background model calls and token spend are on record.", // #254 — two layers: enabled runs the check (audit only), enforce actually @@ -4719,6 +4727,9 @@ window.__ModuleLoader__.load({ // 蒸馏与写入：写进去的东西长什么样——实体抽取（子块：路由与思考强度）、 // 编码回溯、文档摘要卡、质量过滤、原文留存（子块：天数）。 { key: "group.pipeline", items: ["entityExtractionEnabled", "codingRetrospect", "documentMemoryEnabled", "memoryQualityFilter.enabled", + // #687 批内跨类型去重：写侧行为，紧贴质量过滤（同一件事的两道闸—— + // 质量过滤判「值不值得记」，这条判「同一轮里有没有记两遍」）。 + "summarizeBatchCrossTypeDedupe", // #399 的留存总闸放组末：distillSub（天数行）经 subAfter 紧贴它渲染。 "distillRecords.enabled", "imageDescribeEnabled"] }, // 检索与注入：捞什么、排什么序——热记忆、时间前缀、重排、BM25、语义去重、 diff --git a/dsh-mneme/lib/config.js b/dsh-mneme/lib/config.js index 4ab71de5..9cca6e6f 100644 --- a/dsh-mneme/lib/config.js +++ b/dsh-mneme/lib/config.js @@ -96,6 +96,17 @@ export const Config = z.object({ summarizeDedupeMode: z.union([z.const("off"), z.const("title"), z.const("vector")]).default("off"), summarizeDedupeMinSim: z.number().min(0.5).max(0.99).default(0.92), summarizeDedupeWindowHours: z.natural().min(0).max(168).default(24), + // #687 批内跨类型去重（默认开；2026-11-13 桉桉拍板，明知这是行为变化）。同一窗蒸馏 + // 里，模型偶尔把一个结论同时写成 pitfall 与 constraint（或 history 与 project）—— + // 类型不同，summarizeDedupeMode 的 type+title 去重看不见它们。这个闸只认这两组类型对， + // 命中走 supersede（被取代方正文零改写、不 merge）：判错时原文仍在库里，只是不再默认 + // 注入，显式搜索照常可达，继任者被归档时旧行还会自动恢复可见——所以风险不对称， + // 关掉它才是让 #687 报的双录继续产出。不想用时把这一项设 false 即可（面板同一格）。 + // 不开常开类型闸的理由：2026-11-13 实测 45 对里 9 对（20%）合并会丢信息。 + summarizeBatchCrossTypeDedupe: z.boolean().default(true), + // 批内比较的相似度阈（与 quality-filter 的 duplicate 信号同档 0.85：同一条 + // 「是不是在说同一件事」不该有第二个数）。 + summarizeBatchCrossTypeMinSim: z.number().min(0.5).max(0.99).default(0.85), // 智能调速器（429 保护，默认开）：蒸馏 LLM 调用全局串行排队，相邻请求 // 间隔 distillRateLimitIntervalMs（默认 1s 一次）；命中 429 限流时按 // distillRateLimitBaseDelayMs 指数退避（1s→2s→4s…）自动重试 diff --git a/dsh-mneme/lib/entities/extractor.js b/dsh-mneme/lib/entities/extractor.js index be92b0a1..5c6582ff 100644 --- a/dsh-mneme/lib/entities/extractor.js +++ b/dsh-mneme/lib/entities/extractor.js @@ -157,7 +157,12 @@ function sanitizeExtractedData(data, config, logger = null) { } async function resolveEntity(entity, store) { - const existing = store.findEntityByName(entity.name); + // #688: the type rides the lookup. Name-only folding merges genuinely distinct + // objects — the measured library had 95 such groups ("DeepSeek" the technology + // next to "deepseek" the organization). Case/whitespace folding itself happens + // in store.findEntityByName; the extractor must not pre-normalize, or the two + // sides would fold with different rules (JS toLowerCase vs SQLite lower()). + const existing = store.findEntityByName(entity.name, entity.type); if (existing) { store.updateEntity(existing.id, {}); return existing.id; diff --git a/dsh-mneme/lib/lang.js b/dsh-mneme/lib/lang.js index a978a868..0026737e 100644 --- a/dsh-mneme/lib/lang.js +++ b/dsh-mneme/lib/lang.js @@ -15,8 +15,8 @@ const PROMPTS = { "en": "You are the memory curation assistant. From the conversation content below, distill atomic memories worth remembering across sessions.\nAtomic-memory principle: each memory holds exactly one standalone fact/preference/decision — short and self-contained (keep raw details like numbers, names, paths, and conclusions verbatim in content; do not abstract or summarize them away). Prefer splitting into multiple entries over merging and losing detail. Extract 2-4 entries normally, 4-8 for information-dense conversations.\nStance discipline (this wins over the principle above):\n- Record only facts, preferences, decisions, constraints. A preference is the user's stance on a thing — keep recording it. Never record evaluations of the user's own personality, ability, mood or character, favourable or not (\"the user is very smart\", \"the user is frustrated today\" do not belong in memory).\n- The transcript's three roles differ in trustworthiness: \"User:\" is a user statement; \"Assistant:\" and \"Subagent delivery:\" conclusions or evaluations are inferences, not facts. Do not record one as a memory unless the user explicitly adopted it; if it must be kept, say in content that it is an inference. Technical conclusions that tool output, errors or command results state directly are exempt — record those as usual.\n- When a user statement conflicts with an assistant inference, the user statement wins on the facts (never record the assistant inference as fact); but keep both sides as they stand in content — do not write a verdict on who is right and do not adjudicate for the user.\n- importance reflects only how much this fact affects later work. The user's satisfaction, praise or emotion is never a reason to raise it.\nWrite every title and content in English.\nOutput only a JSON array, each item shaped {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"short title\",\"content\":\"one sentence preserving original detail\",\"importance\":1-5,\"occurred_at\":\"optional: absolute time the event happened, ISO-8601\"}.\noccurred_at (optional): the absolute time the described event **actually happened**, in ISO-8601 with a timezone (e.g. 2026-10-05T09:30:00+08:00). When the conversation uses a relative time (yesterday / the day before / last week / last month), convert it into an absolute time against the current time given at the end of this prompt. Omit the field entirely when there is no temporal basis or it cannot be converted — never invent a value, and never pass the current time off as the event time.\n`type` must be one of the enumerated values above (do not invent categories); each item may contain only the listed keys - no id, source or other extra fields.\nOutput no other text." }, "codingSummary": { - "zh": "你是记忆库提炼助手。根据下面的会话内容（含用户输入、助手回答、工具调用与结果），提炼值得跨会话记住的原子记忆。\n原子记忆原则：每条记忆只装一个独立事实/偏好/决策，短小、自带完整上下文（把数字、报错信息、命令、路径、结论等原始细节保留在 content 里，不要抽象概括）；宁可拆成多条也绝不合并丢细节。信息量一般提 2-4 条，信息密集的对话可提 4-8 条。\n立场纪律（与上条原则冲突时以本条为准）：\n- 只记事实、偏好、决策与约束。偏好是用户对某件事的立场，照常记录；但不要记录对用户本人性格、能力、情绪、品行的评价，无论褒贬（「用户很聪明」「用户今天很沮丧」都不进记忆库）。\n- 转录里「用户：」「助手：」「子会话交付：」可信度不同：用户说的话是用户陈述；助手与子会话的结论、评价是推断，不是事实。除非用户明确采纳，否则不要把助手的结论或评价写成记忆；确需保留时，content 里要写明这是推断。工具输出、报错、命令结果直接给出的技术结论不受此限，照常记录。\n- 用户陈述与助手推断冲突时，事实归属以用户陈述为准（不要把助手推断记成事实）；但 content 里要把两种说法如实并存，不要写成谁对谁错、也不要替用户裁决。\n- importance 只看这条事实对后续工作的影响面。用户表达满意、赞许或情绪，都不是提高 importance 的理由。\n只输出 JSON 数组，每项形如 {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"简短标题\",\"content\":\"保留原始细节的一句话\",\"importance\":1-5,\"occurred_at\":\"可选：事件发生的绝对时间，ISO-8601\"}。\noccurred_at（可选）：这条记忆所述事件**实际发生**的绝对时间，ISO-8601 带时区（如 2026-10-05T09:30:00+08:00）。会话里出现「昨天/前天/上周/上个月」这类相对时间时，按提示末尾给出的当前时间换算成绝对时间再填；没有任何时间依据、或换算不出来时，整条省略该字段——严禁编造，也不要把当前时间当成事件时间填进去。\n若对话涉及编码/调试，可额外提取编码类记忆：\n- rejected_solution：被否决/废弃的实现方案（content 含方案简述 + 被否决原因 + 最终采用方案）\n- pitfall：调试踩坑记录（content 含现象/报错 + 根因 + 解决/规避方法）\n- constraint：项目工程约束（content 含约束描述 + 来源）\n普通闲聊、临时无关对话一律不提取编码类记忆。不要输出任何其他文字。", - "en": "You are the memory curation assistant. From the conversation content below (including user input, assistant replies, tool calls and results), distill atomic memories worth remembering across sessions.\nAtomic-memory principle: each memory holds exactly one standalone fact/preference/decision — short and self-contained (keep raw details like numbers, error messages, commands, paths, and conclusions verbatim in content; do not abstract them away). Prefer splitting into multiple entries over merging and losing detail. Extract 2-4 entries normally, 4-8 for information-dense conversations.\nStance discipline (this wins over the principle above):\n- Record only facts, preferences, decisions, constraints. A preference is the user's stance on a thing — keep recording it. Never record evaluations of the user's own personality, ability, mood or character, favourable or not (\"the user is very smart\", \"the user is frustrated today\" do not belong in memory).\n- The transcript's three roles differ in trustworthiness: \"User:\" is a user statement; \"Assistant:\" and \"Subagent delivery:\" conclusions or evaluations are inferences, not facts. Do not record one as a memory unless the user explicitly adopted it; if it must be kept, say in content that it is an inference. Technical conclusions that tool output, errors or command results state directly are exempt — record those as usual.\n- When a user statement conflicts with an assistant inference, the user statement wins on the facts (never record the assistant inference as fact); but keep both sides as they stand in content — do not write a verdict on who is right and do not adjudicate for the user.\n- importance reflects only how much this fact affects later work. The user's satisfaction, praise or emotion is never a reason to raise it.\nWrite every title and content in English.\nOutput only a JSON array, each item shaped {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"short title\",\"content\":\"one sentence preserving original detail\",\"importance\":1-5,\"occurred_at\":\"optional: absolute time the event happened, ISO-8601\"}.\noccurred_at (optional): the absolute time the described event **actually happened**, in ISO-8601 with a timezone (e.g. 2026-10-05T09:30:00+08:00). When the conversation uses a relative time (yesterday / the day before / last week / last month), convert it into an absolute time against the current time given at the end of this prompt. Omit the field entirely when there is no temporal basis or it cannot be converted — never invent a value, and never pass the current time off as the event time.\nIf the conversation involves coding/debugging, additionally extract coding-specific memories:\n- rejected_solution: implementation approaches that were rejected or abandoned (content: brief approach + why rejected + what was adopted instead)\n- pitfall: debugging pitfalls (content: symptom/error + root cause + fix or workaround)\n- constraint: project engineering constraints (content: constraint description + source)\nNever extract coding-specific memories from casual small talk or temporary unrelated conversations. Output no other text." + "zh": "你是记忆库提炼助手。根据下面的会话内容（含用户输入、助手回答、工具调用与结果），提炼值得跨会话记住的原子记忆。\n原子记忆原则：每条记忆只装一个独立事实/偏好/决策，短小、自带完整上下文（把数字、报错信息、命令、路径、结论等原始细节保留在 content 里，不要抽象概括）；宁可拆成多条也绝不合并丢细节。信息量一般提 2-4 条，信息密集的对话可提 4-8 条。\n立场纪律（与上条原则冲突时以本条为准）：\n- 只记事实、偏好、决策与约束。偏好是用户对某件事的立场，照常记录；但不要记录对用户本人性格、能力、情绪、品行的评价，无论褒贬（「用户很聪明」「用户今天很沮丧」都不进记忆库）。\n- 转录里「用户：」「助手：」「子会话交付：」可信度不同：用户说的话是用户陈述；助手与子会话的结论、评价是推断，不是事实。除非用户明确采纳，否则不要把助手的结论或评价写成记忆；确需保留时，content 里要写明这是推断。工具输出、报错、命令结果直接给出的技术结论不受此限，照常记录。\n- 用户陈述与助手推断冲突时，事实归属以用户陈述为准（不要把助手推断记成事实）；但 content 里要把两种说法如实并存，不要写成谁对谁错、也不要替用户裁决。\n- importance 只看这条事实对后续工作的影响面。用户表达满意、赞许或情绪，都不是提高 importance 的理由。\n只输出 JSON 数组，每项形如 {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"简短标题\",\"content\":\"保留原始细节的一句话\",\"importance\":1-5,\"occurred_at\":\"可选：事件发生的绝对时间，ISO-8601\"}。\noccurred_at（可选）：这条记忆所述事件**实际发生**的绝对时间，ISO-8601 带时区（如 2026-10-05T09:30:00+08:00）。会话里出现「昨天/前天/上周/上个月」这类相对时间时，按提示末尾给出的当前时间换算成绝对时间再填；没有任何时间依据、或换算不出来时，整条省略该字段——严禁编造，也不要把当前时间当成事件时间填进去。\n若对话涉及编码/调试，可额外提取编码类记忆：\n- rejected_solution：被否决/废弃的实现方案（content 含方案简述 + 被否决原因 + 最终采用方案）\n- pitfall：调试踩坑记录（content 含现象/报错 + 根因 + 解决/规避方法）\n- constraint：项目工程约束（content 含约束描述 + 来源）\n同一结论只记一条：若已在 pitfall 写了现象/报错 + 根因 + 规避方法，就不要再单开一条同义 constraint（反之亦然）；时间序列或状态快照类内容按时间顺序写进同一条，不要拆成两条同义条目。\n普通闲聊、临时无关对话一律不提取编码类记忆。不要输出任何其他文字。", + "en": "You are the memory curation assistant. From the conversation content below (including user input, assistant replies, tool calls and results), distill atomic memories worth remembering across sessions.\nAtomic-memory principle: each memory holds exactly one standalone fact/preference/decision — short and self-contained (keep raw details like numbers, error messages, commands, paths, and conclusions verbatim in content; do not abstract them away). Prefer splitting into multiple entries over merging and losing detail. Extract 2-4 entries normally, 4-8 for information-dense conversations.\nStance discipline (this wins over the principle above):\n- Record only facts, preferences, decisions, constraints. A preference is the user's stance on a thing — keep recording it. Never record evaluations of the user's own personality, ability, mood or character, favourable or not (\"the user is very smart\", \"the user is frustrated today\" do not belong in memory).\n- The transcript's three roles differ in trustworthiness: \"User:\" is a user statement; \"Assistant:\" and \"Subagent delivery:\" conclusions or evaluations are inferences, not facts. Do not record one as a memory unless the user explicitly adopted it; if it must be kept, say in content that it is an inference. Technical conclusions that tool output, errors or command results state directly are exempt — record those as usual.\n- When a user statement conflicts with an assistant inference, the user statement wins on the facts (never record the assistant inference as fact); but keep both sides as they stand in content — do not write a verdict on who is right and do not adjudicate for the user.\n- importance reflects only how much this fact affects later work. The user's satisfaction, praise or emotion is never a reason to raise it.\nWrite every title and content in English.\nOutput only a JSON array, each item shaped {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"short title\",\"content\":\"one sentence preserving original detail\",\"importance\":1-5,\"occurred_at\":\"optional: absolute time the event happened, ISO-8601\"}.\noccurred_at (optional): the absolute time the described event **actually happened**, in ISO-8601 with a timezone (e.g. 2026-10-05T09:30:00+08:00). When the conversation uses a relative time (yesterday / the day before / last week / last month), convert it into an absolute time against the current time given at the end of this prompt. Omit the field entirely when there is no temporal basis or it cannot be converted — never invent a value, and never pass the current time off as the event time.\nIf the conversation involves coding/debugging, additionally extract coding-specific memories:\n- rejected_solution: implementation approaches that were rejected or abandoned (content: brief approach + why rejected + what was adopted instead)\n- pitfall: debugging pitfalls (content: symptom/error + root cause + fix or workaround)\n- constraint: project engineering constraints (content: constraint description + source)\nRecord each conclusion once: if pitfall already carries the symptom/error + root cause + fix, do not open a synonymous constraint (and vice versa); keep a time series or state snapshot in one entry in time order instead of splitting it into two synonymous entries.\nNever extract coding-specific memories from casual small talk or temporary unrelated conversations. Output no other text." }, "dreamSummary": { "zh": "你是记忆库状态叙述助手。根据整理后的记忆，写一段 150-200 字的「当前状态」叙述：现在在做什么（活跃主题）、最近完成了什么或发生了什么变化、接下来明显的走向。只陈述记忆中有依据的事实，不要发明细节；偏好、项目、决策自然融入叙述而不必逐项罗列。叙述纪律：只写记忆里已写明的事实与偏好；不要给用户添上性格、能力、情绪方面的评价，也不要把助手自己的判断写成「用户认为」。某条记忆本身带褒贬口气时，只取它的事实内核，不要把那个口气复述进叙述。这段叙述会作为会话上下文常驻注入。只输出叙述文本，不要其他内容。", diff --git a/dsh-mneme/lib/service.js b/dsh-mneme/lib/service.js index 35f5ecaf..d0b944e9 100644 --- a/dsh-mneme/lib/service.js +++ b/dsh-mneme/lib/service.js @@ -3403,7 +3403,7 @@ export function createService({ store, mirror, config, onWrite, logger, document }, saveAttr: (r) => store.saveAttr(r), createEntity: (r) => store.createEntity(r), - findEntityByName: (n) => store.findEntityByName(n), + findEntityByName: (n, t) => store.findEntityByName(n, t), findEntityById: (id) => store.findEntityById(id), getAttrsByMemory: (id) => store.getAttrsByMemory(id), // 记忆详情侧栏：一条记忆关联到的实体（entity_attrs.memory_id 反查，纯读）。 diff --git a/dsh-mneme/lib/settings.js b/dsh-mneme/lib/settings.js index d6e57bf9..ea1124cf 100644 --- a/dsh-mneme/lib/settings.js +++ b/dsh-mneme/lib/settings.js @@ -195,6 +195,10 @@ const FEATURE_FLAG_BOOLEANS = [ "reflectionUpdateEnabled", "reflectionFailureTracking", "bm25SearchEnabled", + // #687：批内跨类型去重（默认关 = 现状）。同一窗蒸馏里模型偶尔把一条结论同时写成 + // pitfall 与 constraint（或 history 与 project），类型不同，summarizeDedupeMode 的 + // type+title 去重看不见它们。开后可回面板调阈值；判错时被取代方正文仍在库里。 + "summarizeBatchCrossTypeDedupe", "conflictFreezeEnabled", // Issue #603 第 4 层：自动裁决档（默认关）。相似度极高的同 scope 重复对直接 // 保留新条、归档旧条，不进人工队列；模糊地带照旧留给人。会归档记忆，所以必须 @@ -392,6 +396,8 @@ const FEATURE_FLAG_INT_RANGES = { const FEATURE_FLAG_NUMBER_RANGES = { // Issue #127：vector 去重档的并入阈值。 summarizeDedupeMinSim: [0.5, 0.99], + // #687：批内跨类型去重的相似度阈（与 config.js 的 min/max 成对）。 + summarizeBatchCrossTypeMinSim: [0.5, 0.99], // Issue #125：hybrid 判"高相似"的阈值。 dreamCandidateMinSim: [0.5, 0.99], // Issue #24 块2：单次有效触达的边权抬升幅度（与 config.js 的闭区间对齐）。 diff --git a/dsh-mneme/lib/store.js b/dsh-mneme/lib/store.js index 4b72c5f5..b1f77992 100644 --- a/dsh-mneme/lib/store.js +++ b/dsh-mneme/lib/store.js @@ -540,11 +540,16 @@ CREATE TABLE IF NOT EXISTS entities ( type TEXT, first_seen TEXT NOT NULL, last_seen TEXT NOT NULL, - mention_count INTEGER DEFAULT 1, - canonical_memory_id TEXT + mention_count INTEGER DEFAULT 1 ); CREATE INDEX IF NOT EXISTS idx_entities_name ON entities(name); CREATE INDEX IF NOT EXISTS idx_entities_type ON entities(type); +-- #688: 写入侧归一的查找键是 lower(trim(name)) + type（见 findEntityByName）。表达式索引 +-- 让这条查询不必扫全表——本机实体表 25,474 行，实体抽取是每条记忆都要走的写入热路径。 +-- 折叠在 SQL 里做（不是先在 JS 里折好再绑），是为了让「命中键」与「索引键」逐字节同源： +-- JS 的 toLowerCase 与 SQLite 的 lower() 只在 ASCII 上一致，混用会在含特殊字符的名字上 +-- 造出一行查不到自己的孤儿。存量行不改写，索引只是给旧值补一个可查的键。 +CREATE INDEX IF NOT EXISTS idx_entities_name_key ON entities(lower(trim(name)), type); CREATE TABLE IF NOT EXISTS entity_attrs ( id TEXT PRIMARY KEY, @@ -1215,8 +1220,7 @@ function toEntity(row) { type: row.type ?? undefined, first_seen: row.first_seen, last_seen: row.last_seen, - mention_count: row.mention_count, - canonical_memory_id: row.canonical_memory_id ?? undefined + mention_count: row.mention_count }; } @@ -1479,6 +1483,29 @@ export function createStore(path) { // 行缺列 → 默认 manual 1.0（最保守：既有语义不因迁移而变弱）。 addColumn("entity_relations", "weight", "ALTER TABLE entity_relations ADD COLUMN weight REAL NOT NULL DEFAULT 1.0"); addColumn("entity_relations", "source", "ALTER TABLE entity_relations ADD COLUMN source TEXT NOT NULL DEFAULT 'manual'"); + + // #688：entities.canonical_memory_id 是 v0.7.1 预留的「规范行指针」，此后长成了 + // 一根孤儿——全仓只有 plumbing（DDL、行映射、INSERT、patch、UPDATE），没有一处 SELECT + // 或业务写入，真实库里 25,474 行全是 NULL。留着它不是中性的：下一位读者会把它当成 + // 一条可用的判据（#688 的实体归并讨论里就差点被当成「同一条记忆的实体该并」）。 + // 删列没有数据要保（本来就全空），也不需要 applied_migrations 登记——判定条件是 + // 「列还在不在」，跑完恒不命中，和 addColumn 同一族。 + // + // 前提：SQLite >= 3.35 才有 DROP COLUMN（本项目 node:sqlite 在支持的运行时上是 + // 3.46+，本机 3.51.3）。竞态与 addColumn 同理：两个进程可能同时通过检查，后到的 + // 那个收到的是 "no such column"，那是预期结局而不是错误。 + const dropColumn = (table, column) => { + const cols = db.prepare(`PRAGMA table_info(${table})`).all().map((c) => c.name); + if (!cols.includes(column)) return false; + try { + db.exec(`ALTER TABLE ${table} DROP COLUMN ${column}`); + return true; + } catch (e) { + if (/no such column/i.test(String(e?.message ?? e))) return false; + throw e; + } + }; + dropColumn("entities", "canonical_memory_id"); // 列序是 content_hash 打头：哈希几乎唯一，等值 seek 就已收窄到候选行，type 只当同 // 一次 seek 里的第二列过滤；反过来（type 打头）下面的存量回填就得全表扫——每次打开 // 都要把全部正文读一遍。 @@ -4685,15 +4712,49 @@ export function createStore(path) { function createEntity({ name, type }) { const id = randomUUID(); const now = nowIso(); + // 落库前只去掉首尾空白，保留原大小写：显示名是给人看的（"DeepSeek" 不该被写成 + // "deepseek"），大小写折叠只发生在查找键上。存量行按 #688 口径一行不动。 + const storedName = String(name ?? "").trim(); db.prepare( - `INSERT INTO entities (id, name, type, first_seen, last_seen, mention_count, canonical_memory_id) - VALUES (?, ?, ?, ?, ?, ?, ?)` - ).run(id, name, type ?? null, now, now, 1, null); + `INSERT INTO entities (id, name, type, first_seen, last_seen, mention_count) + VALUES (?, ?, ?, ?, ?, ?)` + ).run(id, storedName, type ?? null, now, now, 1); return toEntity(db.prepare("SELECT * FROM entities WHERE id = ?").get(id)); } - function findEntityByName(name) { - return toEntity(db.prepare("SELECT * FROM entities WHERE name = ?").get(name)); + /** + * Resolve an entity by its normalized key: lower(trim(name)), optionally + * narrowed to one type. + * + * #688: the pre-fix lookup was `WHERE name = ?` — byte equality. "DeepSeek" and + * "deepseek " were two rows forever, even though the extractor means the same + * object; the measured stock was 170 same-type groups collapsed by this fold. + * + * Why the type dimension is not optional in the write path: folding name alone + * merges 95 groups that are genuinely different objects — "DeepSeek" the + * technology vs "deepseek" the organization is the canonical example. Callers + * that resolve a *mention* must pass the type; callers that look up by a + * user-supplied name (panel search) may omit it and get the most recently seen + * row across types. + * + * Why no version-suffix stripping: "DeepSeek-V3" and "DeepSeek-V4" are distinct + * models; collapsing them loses which one a constraint is about. + * + * @param {string} name + * @param {string|null} [type] entity type; when undefined/null the lookup is + * type-agnostic (panel / entity-name search path) + * @returns {object|undefined} + */ + function findEntityByName(name, type) { + if (name === undefined || name === null) return undefined; + const hasType = type !== undefined && type !== null; + const sql = hasType + ? `SELECT * FROM entities WHERE lower(trim(name)) = lower(trim(?)) AND type IS ? + ORDER BY last_seen DESC LIMIT 1` + : `SELECT * FROM entities WHERE lower(trim(name)) = lower(trim(?)) + ORDER BY last_seen DESC LIMIT 1`; + const params = hasType ? [String(name), String(type)] : [String(name)]; + return toEntity(db.prepare(sql).get(...params)); } function findEntityById(id) { @@ -4711,16 +4772,13 @@ export function createStore(path) { const has = (k) => Object.prototype.hasOwnProperty.call(patch, k); const name = has("name") ? patch.name : old.name; const type = has("type") ? patch.type : old.type; - const canonical_memory_id = has("canonical_memory_id") - ? patch.canonical_memory_id - : old.canonical_memory_id; const mention_count = has("mention_count") ? patch.mention_count : (old.mention_count ?? 1) + 1; const now = nowIso(); db.prepare( - `UPDATE entities SET name = ?, type = ?, last_seen = ?, mention_count = ?, canonical_memory_id = ? WHERE id = ?` - ).run(name, type ?? null, now, mention_count, canonical_memory_id ?? null, id); + `UPDATE entities SET name = ?, type = ?, last_seen = ?, mention_count = ? WHERE id = ?` + ).run(name, type ?? null, now, mention_count, id); return findEntityById(id); } diff --git a/dsh-mneme/lib/summarize.js b/dsh-mneme/lib/summarize.js index 7e4f1d73..dbb1aab0 100644 --- a/dsh-mneme/lib/summarize.js +++ b/dsh-mneme/lib/summarize.js @@ -17,6 +17,10 @@ import { contentToText, countImages, IMAGE_PLACEHOLDER_PREFIX } from "./attachme // 图像描述档（docs/DISTILL.md:346）：占位之外再补一段「图里是什么」。本文件只 // 负责把描述并进转录行；出描述的调用、准入判断都在 image-describe.js。 import { appendDescriptions, describeImages, imageBlocks, canSeeImages, DESCRIPTION_PREFIX, imageRefOf } from "./image-describe.js"; +// #687：批内跨类型去重沿用写入侧质量过滤那把相似度（同一份归一化 bigram 重叠、 +// 同一族阈值），不另造第二套度量——两套度量会给出两套「像不像」的答案，而这一条 +// 判据只该有一个。 +import { textSimilarity } from "./quality-filter.js"; // #512：每轮蒸馏一个 run id（归因主体）。store 侧也能铸造，但这里必须知道它—— // 写记忆时要把它一路带到 saveMemoryProvenance / 落 run 行，拿回来再写就是两次往返。 import { randomUUID } from "node:crypto"; @@ -38,6 +42,87 @@ const OCCURRED_AT_FUTURE_TOLERANCE_MS = 24 * 60 * 60 * 1000; // 刻意不做成配置键：护栏本身默认关（summarizeParseFailureMaxRetries=0），一个只为 // 护栏服务的失效窗口不该再占一档用户可见配置（AGENTS.md 旗标计数锁也要跟着动）。 const PARSE_FAILURE_STREAK_STALE_MS = 7 * 24 * 60 * 60 * 1000; + +// --- #687 批内跨类型去重 --------------------------------------------------------- +// 一次蒸馏里，同一个结论常被写成两条不同类型的记忆（"踩坑" 与 "约束" 其实是同一句 +// 话；一次事件与它所属的项目状态写了两遍）。2026-11-13 在真实库上量过：跨类型余弦 +// >= 0.90 的 45 对里 36 对（80%）同义，集中在这两组类型对上（constraint<->pitfall 9、 +// history<->project 8）。但同一批数据里还有 9 对（20%）合并会丢信息：5 对是时间序列 +// /状态快照、2 对是「症状」与「修法」、1 对结论相反、1 对版本号不同 —— 所以不开 +// 常开的类型闸，只在**同一批**（同一窗蒸馏、同一件事）上认这两组。 +export const BATCH_CROSS_TYPE_PAIRS = Object.freeze([ + Object.freeze(["pitfall", "constraint"]), + Object.freeze(["history", "project"]) +]); +// 与 quality-filter 的 duplicate 信号同一档（0.85）：同一条判据「这两段是不是在说 +// 同一件事」在写入侧已经用这个数，这里再拍一个相近的数只会让两处阈值互相漂移。 +export const BATCH_CROSS_TYPE_MIN_SIM = 0.85; +// 下面两条不是阈值旋钮，是「相似」与「同一结论」之间的分隔线： +// - 太短的条目不参与：textSimilarity 以较小一侧为分母，一句 15 字的话几乎完全被任何 +// 长段落包含 → 相似度接近 1，但那是「被提到」而不是「重复」； +// - 长度悬殊的条目也不参与（短边不到长边一半）：那是包含关系（一段长记录里顺手提了 +// 另一个结论），合并它正是在丢信息。 +const BATCH_CROSS_TYPE_MIN_CHARS = 24; +const BATCH_CROSS_TYPE_MIN_LEN_RATIO = 0.5; + +function batchPairKey(a, b) { + for (const [left, right] of BATCH_CROSS_TYPE_PAIRS) { + if ((a === left && b === right) || (a === right && b === left)) return `${left}<->${right}`; + } + return null; +} + +/** + * Plan the in-batch cross-type supersede pass over one distillation window's + * entries. Pure: no I/O, no ids — it returns indices, because the rows do not + * exist yet (the caller chains them after writing). + * + * Pairs are visited by descending similarity so the strongest evidence wins the + * right to retire an entry; an entry is retired at most once and may not retire + * another entry that is itself retired (no chains that would hide a row behind a + * row that is already hidden). + * + * Winner = the entry carrying more content; equal length falls back to the entry + * written earlier in the batch (lower index). "More content" rather than "newer" + * because both were produced by the same call — there is no recency to speak of, + * and the fuller text is what a supersede should preserve. + * + * @param {Array<{type?:string,title?:string,content?:string}>} entries + * @param {{minSim?:number}} [options] + * @returns {Array<{winnerIndex:number, loserIndex:number, sim:number, pair:string}>} + */ +export function planBatchCrossTypeDedupe(entries, { minSim = BATCH_CROSS_TYPE_MIN_SIM } = {}) { + const plans = []; + if (!Array.isArray(entries) || entries.length < 2) return plans; + const candidates = []; + for (let i = 0; i < entries.length; i++) { + for (let j = i + 1; j < entries.length; j++) { + const a = entries[i] ?? {}; + const b = entries[j] ?? {}; + const pair = batchPairKey(a.type, b.type); + if (!pair) continue; + const la = String(a.content ?? "").length; + const lb = String(b.content ?? "").length; + const shortSide = Math.min(la, lb); + const longSide = Math.max(la, lb); + if (shortSide < BATCH_CROSS_TYPE_MIN_CHARS) continue; + if (shortSide / longSide < BATCH_CROSS_TYPE_MIN_LEN_RATIO) continue; + const sim = textSimilarity(`${a.title ?? ""}\n${a.content ?? ""}`, `${b.title ?? ""}\n${b.content ?? ""}`); + if (sim < minSim) continue; + const winnerIndex = la >= lb ? i : j; + const loserIndex = winnerIndex === i ? j : i; + candidates.push({ winnerIndex, loserIndex, sim, pair }); + } + } + candidates.sort((x, y) => y.sim - x.sim || x.loserIndex - y.loserIndex); + const retired = new Set(); + for (const c of candidates) { + if (retired.has(c.loserIndex) || retired.has(c.winnerIndex)) continue; + retired.add(c.loserIndex); + plans.push({ winnerIndex: c.winnerIndex, loserIndex: c.loserIndex, sim: Number(c.sim.toFixed(4)), pair: c.pair }); + } + return plans; +} function normalizeDistilledOccurredAt(raw, nowMs = Date.now()) { if (typeof raw !== "string") return undefined; const text = raw.trim(); @@ -1233,6 +1318,10 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = const dedupeMode = config.summarizeDedupeMode ?? "off"; let deduped = 0; let dedupeMaxSim = 0; + // #687：批内跨类型去重的落账计数（链成功几对）。默认关着一律为 0， + // 审计行与加它之前逐字节一致。 + let batchDeduped = 0; + const batchChainedPairs = []; // v0.9.0（M.O. 项目记忆）：一次窗口解析一次——同一蒸馏窗口里所有条目同属一个 // 工作区、一个 agent 身份，逐条解析是白花钱。解析不到就落 NULL（NULL = 未标注 = // 全局可见），且解析器绝不抛错（scope.js 的硬约束：记忆写入不该被作用域拖挂）。 @@ -1269,6 +1358,31 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = } writes.push({ entry, source, dup }); } + // #687：批内跨类型去重（opt-in，默认关 = 现状）。同一窗蒸馏里模型常把一个结论 + // 写成两条不同类型（pitfall 与 constraint 同义、history 与 project 同义）， + // 而"类型不同"恰好让写入侧的 type+title 去重看不见它们。合并方向固定为 + // supersede：正文零改写、不 merge —— 判错时原文仍在库里，只是默认注入不再取它， + // 而 merge 会真的抹掉一条的措辞，没有回头路。 + // + // 已被并进既有行的条目（dup 有值）不参与：它的正文已经进了那条老记忆，把老记忆 + // 盖到下一条新条目下面，等于用一次蒸馏顺手隐藏一条累积了很久的旧行。 + const batchPlans = []; + if (config?.summarizeBatchCrossTypeDedupe !== false && writes.length > 1) { + const plannable = writes + .map((write, index) => ({ entry: write.entry, index })) + .filter((slot) => !writes[slot.index].dup); + const planned = planBatchCrossTypeDedupe(plannable.map((slot) => slot.entry), { + minSim: config?.summarizeBatchCrossTypeMinSim ?? BATCH_CROSS_TYPE_MIN_SIM + }); + for (const plan of planned) { + batchPlans.push({ + winnerIndex: plannable[plan.winnerIndex].index, + loserIndex: plannable[plan.loserIndex].index, + sim: plan.sim, + pair: plan.pair + }); + } + } // #399：蒸馏窗口的原始公开记录 + 记忆级溯源链接，与记忆写入同事务落盘。 // 凭证与它的派生记忆必须同生共死：记忆写了而记录没写，溯源就断在「不知道 // 出自哪段原文」；记录写了而记忆没写，盘上留一份没有索引的原文。默认关 @@ -1280,7 +1394,9 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = : null; const writeWindow = () => { const savedIds = []; - for (const { entry, source, dup } of writes) { + // #687：批内 supersede 需要「写之后」才有的 id，按 writes 的下标记住它们。 + const idByWriteIndex = new Map(); + for (const [writeIndex, { entry, source, dup }] of writes.entries()) { const outcome = service.saveWithDedupe({ ...entry, source, @@ -1301,6 +1417,19 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = // created 与 merged 两条成功分支都返回 memory（含 id）；merged 时 id 是 // 被并入的既有记忆——本次窗口的原文确实贡献了它，链接它是诚实的。 if (outcome?.memory?.id) savedIds.push(outcome.memory.id); + if (outcome?.memory?.id) idByWriteIndex.set(writeIndex, outcome.memory.id); + } + // #687：两条都落库之后才链——supersedeChain 写的是行间指针，先链后写会指向 + // 一个还不存在的 id。链失败（行被并发归档/物理删除）就跳过：这一对的正文都 + // 已经写进去了，宁可留着重复，也不能为了「看起来干净」去伪造一次链。 + for (const plan of batchPlans) { + const loserId = idByWriteIndex.get(plan.loserIndex); + const winnerId = idByWriteIndex.get(plan.winnerIndex); + if (!loserId || !winnerId || loserId === winnerId) continue; + if (service.supersedeChain?.(loserId, winnerId)) { + batchDeduped++; + batchChainedPairs.push(plan.pair); + } } // #512：run 的条数口径是「本次写成功的记忆行数」，与 savedIds 同源—— // 另起一个计数器会在 merged 分支上与它对不上（那一路也 push 了 id）。 @@ -1401,7 +1530,11 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = parsed: parsed.length, ...(parsedResult.salvaged ? { json_salvaged: true } : {}), ...(capped > 0 ? { capped } : {}), - ...(deduped > 0 ? { deduped, mode: dedupeMode, maxSim: Number(dedupeMaxSim.toFixed(4)) } : {}) + ...(deduped > 0 ? { deduped, mode: dedupeMode, maxSim: Number(dedupeMaxSim.toFixed(4)) } : {}), + // #687：批内跨类型 supersede 的落账（几对、为什么链）。默认关时这个键根本不出现。 + ...(batchDeduped > 0 + ? { batchDeduped, batchPairs: batchChainedPairs } + : {}) }; } // 记忆写入和解析都成功后才提交窗口；流失败、中止、解析失败或写入异常 diff --git a/dsh-mneme/scripts/cross-type-dedupe-dryrun.mjs b/dsh-mneme/scripts/cross-type-dedupe-dryrun.mjs new file mode 100644 index 00000000..a1065d52 --- /dev/null +++ b/dsh-mneme/scripts/cross-type-dedupe-dryrun.mjs @@ -0,0 +1,232 @@ +#!/usr/bin/env node +// #687 one-time stock take: list cross-type near-duplicate memory pairs that the +// batch-level rule would have collapsed, so a human can approve a supersede pass +// before anything widens. READ-ONLY by construction: the database is opened with +// readOnly: true and this script contains no INSERT / UPDATE / DELETE — the only +// thing it ever produces is a report. +// +// Why only two type pairs: the 2026-11-13 measurement over the live library found +// 45 pairs at cosine >= 0.90, of which 36 (80%) were genuinely synonymous but 9 +// merged away real information (5 time-series/state-snapshot, 2 symptom-vs-fix, +// 1 inverted conclusion, 1 different version). The concentration was in +// constraint<->pitfall (9) and history<->project (8), so the in-batch rule is +// gated to exactly those two pairs rather than opening a permanent type gate. +// +// Usage: +// node scripts/cross-type-dedupe-dryrun.mjs [--db ] [--min-sim 0.9] [--out ] +// +// The report contains memory titles/content — private data. The default --out is +// outside the repository and an --out inside the repository is refused, because +// this file must never be committed. + +import { DatabaseSync } from "node:sqlite"; +import { writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve, sep } from "node:path"; +import { fileURLToPath } from "node:url"; +import { textSimilarity } from "../src/quality-filter.js"; + +const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..", ".."); + +// The two pairs the rule is allowed to act on. Everything else is out of scope, +// including same-type duplicates (handled by saveWithDedupe / autoDream). +const ALLOWED_PAIRS = [ + ["pitfall", "constraint"], + ["history", "project"] +]; + +function parseArgs(argv) { + const args = { db: null, minSim: 0.9, out: null, json: false }; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a === "--db") args.db = argv[++i]; + else if (a === "--min-sim") args.minSim = Number(argv[++i]); + else if (a === "--out") args.out = argv[++i]; + else if (a === "--json") args.json = true; + else if (a === "--help" || a === "-h") { + console.log("usage: node scripts/cross-type-dedupe-dryrun.mjs [--db ] [--min-sim 0.9] [--out ] [--json]"); + process.exit(0); + } else { + throw new Error(`unknown argument: ${a}`); + } + } + return args; +} + +function defaultDbPath() { + const home = process.env.DSH_HOME || join(process.env.USERPROFILE || process.env.HOME || ".", ".dsh"); + return join(home, "memory", "memory.db"); +} + +function parseVector(raw) { + if (typeof raw !== "string" || !raw) return null; + let arr; + try { + arr = JSON.parse(raw); + } catch { + return null; + } + if (!Array.isArray(arr) || arr.length === 0) return null; + const vec = new Float64Array(arr.length); + let norm = 0; + for (let i = 0; i < arr.length; i++) { + const v = Number(arr[i]); + if (!Number.isFinite(v)) return null; + vec[i] = v; + norm += v * v; + } + if (norm <= 0) return null; + const inv = 1 / Math.sqrt(norm); + for (let i = 0; i < vec.length; i++) vec[i] *= inv; + return vec; +} + +function cosine(a, b) { + // Dimension mismatch is not a near-miss, it is an unanswerable comparison: + // return null so the caller can count it instead of scoring it 0 (the silent + // "runs to completion, looks fine" failure this project already ate once). + if (!a || !b || a.length !== b.length) return null; + let s = 0; + for (let i = 0; i < a.length; i++) s += a[i] * b[i]; + return s; +} + +// "Keep the fuller one" must be a deterministic function of the row, not of +// iteration order: content length first (the whole reason for superseding +// instead of merging), then the later edit, then id as the final tiebreak. +function pickWinner(a, b) { + const la = String(a.content ?? "").length; + const lb = String(b.content ?? "").length; + if (la !== lb) return la > lb ? { winner: a, loser: b, criterion: "longer-content" } : { winner: b, loser: a, criterion: "longer-content" }; + const ua = String(a.updated_at ?? ""); + const ub = String(b.updated_at ?? ""); + if (ua !== ub) return ua > ub ? { winner: a, loser: b, criterion: "newer-updated_at" } : { winner: b, loser: a, criterion: "newer-updated_at" }; + return a.id < b.id ? { winner: a, loser: b, criterion: "id-tiebreak" } : { winner: b, loser: a, criterion: "id-tiebreak" }; +} + +const args = parseArgs(process.argv.slice(2)); +const dbPath = resolve(args.db ?? defaultDbPath()); +const outPath = resolve(args.out ?? join(tmpdir(), "mneme-cross-type-dedupe-dryrun.json")); + +// Guard the guard: a report holding memory text inside the checkout would end up +// in a commit sooner or later. Refuse instead of relying on the operator. +if (outPath === REPO_ROOT || outPath.startsWith(REPO_ROOT + sep)) { + throw new Error(`refusing to write the report inside the repository (${outPath}); pass --out to a path outside ${REPO_ROOT}`); +} + +const db = new DatabaseSync(dbPath, { readOnly: true }); +db.exec("PRAGMA busy_timeout = 5000;"); + +const types = [...new Set(ALLOWED_PAIRS.flat())]; +const placeholders = types.map(() => "?").join(", "); +const rows = db.prepare( + `SELECT id, type, title, content, updated_at, embedding + FROM memories + WHERE type IN (${placeholders}) + AND archived = 0 + AND forgotten = 0 + AND superseded_by IS NULL + AND embedding IS NOT NULL` +).all(...types); + +const byType = new Map(types.map((t) => [t, []])); +let unvectorized = 0; +let dimMismatch = 0; +for (const row of rows) { + row.vec = parseVector(row.embedding); + delete row.embedding; + if (!row.vec) { + unvectorized++; + continue; + } + byType.get(row.type)?.push(row); +} + +const dims = new Set([...byType.values()].flat().map((r) => r.vec.length)); +if (dims.size > 1) { + // Two embedding generations in one library means cross-generation cosines are + // meaningless; say so instead of producing a number nobody should trust. + throw new Error(`library holds mixed embedding dimensions (${[...dims].join(", ")}); refusing to compare across them`); +} + +const pairs = []; +for (const [left, right] of ALLOWED_PAIRS) { + const A = byType.get(left) ?? []; + const B = byType.get(right) ?? []; + for (const a of A) { + for (const b of B) { + const sim = cosine(a.vec, b.vec); + if (sim === null) { + dimMismatch++; + continue; + } + if (sim < args.minSim) continue; + const pick = pickWinner(a, b); + pairs.push({ + left: a.type, + right: b.type, + leftId: a.id, + rightId: b.id, + cosine: Number(sim.toFixed(4)), + // Second, independent signal: the same bigram overlap the batch rule + // uses, so the reviewer can see which pairs survive a *text* comparison + // too (a high cosine with low text overlap is often a same-topic, + // different-fact pair — exactly the kind the 2026-11-13 audit found). + textSim: Number(textSimilarity(`${a.title}\n${a.content}`, `${b.title}\n${b.content}`).toFixed(4)), + keep: pick.winner.id, + drop: pick.loser.id, + criterion: pick.criterion, + keepType: pick.winner.type, + dropType: pick.loser.type, + keepTitle: pick.winner.title, + dropTitle: pick.loser.title, + keepLen: String(pick.winner.content ?? "").length, + dropLen: String(pick.loser.content ?? "").length + }); + } + } +} + +pairs.sort((x, y) => y.cosine - x.cosine || x.keep.localeCompare(y.keep)); + +const report = { + generatedAt: new Date().toISOString(), + db: dbPath, + readOnly: true, + minSim: args.minSim, + allowedPairs: ALLOWED_PAIRS, + scanned: { [types[0]]: byType.get(types[0])?.length ?? 0, [types[1]]: byType.get(types[1])?.length ?? 0, unvectorized, dimMismatch }, + pairCount: pairs.length, + byPair: Object.fromEntries(ALLOWED_PAIRS.map(([l, r]) => [`${l}<->${r}`, pairs.filter((p) => (p.left === l && p.right === r)).length])), + // The batch rule reuses textSim (bigram overlap) as its gate; the table below + // shows how the two signals line up on real data before that default is set. + textSimBands: Object.fromEntries( + [[0.5, 0.6], [0.6, 0.7], [0.7, 0.8], [0.8, 0.9], [0.9, 1.01]].map(([lo, hi]) => [ + `${lo}-${hi >= 1 ? 1 : hi}`, + pairs.filter((p) => p.textSim >= lo && p.textSim < hi).length + ]) + ), + pairs +}; + +writeFileSync(outPath, JSON.stringify(report, null, 2), "utf8"); + +if (args.json) { + console.log(JSON.stringify(report, null, 2)); +} else { + console.log(`read-only scan: ${dbPath}`); + console.log(`types: ${types.join(", ")}; scanned ${rows.length} rows (unvectorized ${unvectorized}, dim mismatch ${dimMismatch})`); + console.log(`pairs at cosine >= ${args.minSim}: ${pairs.length} (${Object.entries(report.byPair).map(([k, v]) => `${k}=${v}`).join(", ")})`); + console.log("textSim bands:", JSON.stringify(report.textSimBands)); + console.log(""); + console.log("cosine textSim pair keep(id/len) drop(id/len) criterion"); + for (const p of pairs) { + console.log( + `${p.cosine.toFixed(4)} ${p.textSim.toFixed(4)} ${`${p.left}/${p.right}`.padEnd(20)} ${`${p.keep.slice(0, 8)}/${p.keepLen}`.padEnd(14)} ${`${p.drop.slice(0, 8)}/${p.dropLen}`.padEnd(14)} ${p.criterion}` + ); + } + console.log(""); + console.log(`report: ${outPath}`); +} + +db.close(); \ No newline at end of file diff --git a/dsh-mneme/src/config.js b/dsh-mneme/src/config.js index 4ab71de5..9cca6e6f 100644 --- a/dsh-mneme/src/config.js +++ b/dsh-mneme/src/config.js @@ -96,6 +96,17 @@ export const Config = z.object({ summarizeDedupeMode: z.union([z.const("off"), z.const("title"), z.const("vector")]).default("off"), summarizeDedupeMinSim: z.number().min(0.5).max(0.99).default(0.92), summarizeDedupeWindowHours: z.natural().min(0).max(168).default(24), + // #687 批内跨类型去重（默认开；2026-11-13 桉桉拍板，明知这是行为变化）。同一窗蒸馏 + // 里，模型偶尔把一个结论同时写成 pitfall 与 constraint（或 history 与 project）—— + // 类型不同，summarizeDedupeMode 的 type+title 去重看不见它们。这个闸只认这两组类型对， + // 命中走 supersede（被取代方正文零改写、不 merge）：判错时原文仍在库里，只是不再默认 + // 注入，显式搜索照常可达，继任者被归档时旧行还会自动恢复可见——所以风险不对称， + // 关掉它才是让 #687 报的双录继续产出。不想用时把这一项设 false 即可（面板同一格）。 + // 不开常开类型闸的理由：2026-11-13 实测 45 对里 9 对（20%）合并会丢信息。 + summarizeBatchCrossTypeDedupe: z.boolean().default(true), + // 批内比较的相似度阈（与 quality-filter 的 duplicate 信号同档 0.85：同一条 + // 「是不是在说同一件事」不该有第二个数）。 + summarizeBatchCrossTypeMinSim: z.number().min(0.5).max(0.99).default(0.85), // 智能调速器（429 保护，默认开）：蒸馏 LLM 调用全局串行排队，相邻请求 // 间隔 distillRateLimitIntervalMs（默认 1s 一次）；命中 429 限流时按 // distillRateLimitBaseDelayMs 指数退避（1s→2s→4s…）自动重试 diff --git a/dsh-mneme/src/entities/extractor.js b/dsh-mneme/src/entities/extractor.js index be92b0a1..5c6582ff 100644 --- a/dsh-mneme/src/entities/extractor.js +++ b/dsh-mneme/src/entities/extractor.js @@ -157,7 +157,12 @@ function sanitizeExtractedData(data, config, logger = null) { } async function resolveEntity(entity, store) { - const existing = store.findEntityByName(entity.name); + // #688: the type rides the lookup. Name-only folding merges genuinely distinct + // objects — the measured library had 95 such groups ("DeepSeek" the technology + // next to "deepseek" the organization). Case/whitespace folding itself happens + // in store.findEntityByName; the extractor must not pre-normalize, or the two + // sides would fold with different rules (JS toLowerCase vs SQLite lower()). + const existing = store.findEntityByName(entity.name, entity.type); if (existing) { store.updateEntity(existing.id, {}); return existing.id; diff --git a/dsh-mneme/src/lang.js b/dsh-mneme/src/lang.js index a978a868..0026737e 100644 --- a/dsh-mneme/src/lang.js +++ b/dsh-mneme/src/lang.js @@ -15,8 +15,8 @@ const PROMPTS = { "en": "You are the memory curation assistant. From the conversation content below, distill atomic memories worth remembering across sessions.\nAtomic-memory principle: each memory holds exactly one standalone fact/preference/decision — short and self-contained (keep raw details like numbers, names, paths, and conclusions verbatim in content; do not abstract or summarize them away). Prefer splitting into multiple entries over merging and losing detail. Extract 2-4 entries normally, 4-8 for information-dense conversations.\nStance discipline (this wins over the principle above):\n- Record only facts, preferences, decisions, constraints. A preference is the user's stance on a thing — keep recording it. Never record evaluations of the user's own personality, ability, mood or character, favourable or not (\"the user is very smart\", \"the user is frustrated today\" do not belong in memory).\n- The transcript's three roles differ in trustworthiness: \"User:\" is a user statement; \"Assistant:\" and \"Subagent delivery:\" conclusions or evaluations are inferences, not facts. Do not record one as a memory unless the user explicitly adopted it; if it must be kept, say in content that it is an inference. Technical conclusions that tool output, errors or command results state directly are exempt — record those as usual.\n- When a user statement conflicts with an assistant inference, the user statement wins on the facts (never record the assistant inference as fact); but keep both sides as they stand in content — do not write a verdict on who is right and do not adjudicate for the user.\n- importance reflects only how much this fact affects later work. The user's satisfaction, praise or emotion is never a reason to raise it.\nWrite every title and content in English.\nOutput only a JSON array, each item shaped {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"short title\",\"content\":\"one sentence preserving original detail\",\"importance\":1-5,\"occurred_at\":\"optional: absolute time the event happened, ISO-8601\"}.\noccurred_at (optional): the absolute time the described event **actually happened**, in ISO-8601 with a timezone (e.g. 2026-10-05T09:30:00+08:00). When the conversation uses a relative time (yesterday / the day before / last week / last month), convert it into an absolute time against the current time given at the end of this prompt. Omit the field entirely when there is no temporal basis or it cannot be converted — never invent a value, and never pass the current time off as the event time.\n`type` must be one of the enumerated values above (do not invent categories); each item may contain only the listed keys - no id, source or other extra fields.\nOutput no other text." }, "codingSummary": { - "zh": "你是记忆库提炼助手。根据下面的会话内容（含用户输入、助手回答、工具调用与结果），提炼值得跨会话记住的原子记忆。\n原子记忆原则：每条记忆只装一个独立事实/偏好/决策，短小、自带完整上下文（把数字、报错信息、命令、路径、结论等原始细节保留在 content 里，不要抽象概括）；宁可拆成多条也绝不合并丢细节。信息量一般提 2-4 条，信息密集的对话可提 4-8 条。\n立场纪律（与上条原则冲突时以本条为准）：\n- 只记事实、偏好、决策与约束。偏好是用户对某件事的立场，照常记录；但不要记录对用户本人性格、能力、情绪、品行的评价，无论褒贬（「用户很聪明」「用户今天很沮丧」都不进记忆库）。\n- 转录里「用户：」「助手：」「子会话交付：」可信度不同：用户说的话是用户陈述；助手与子会话的结论、评价是推断，不是事实。除非用户明确采纳，否则不要把助手的结论或评价写成记忆；确需保留时，content 里要写明这是推断。工具输出、报错、命令结果直接给出的技术结论不受此限，照常记录。\n- 用户陈述与助手推断冲突时，事实归属以用户陈述为准（不要把助手推断记成事实）；但 content 里要把两种说法如实并存，不要写成谁对谁错、也不要替用户裁决。\n- importance 只看这条事实对后续工作的影响面。用户表达满意、赞许或情绪，都不是提高 importance 的理由。\n只输出 JSON 数组，每项形如 {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"简短标题\",\"content\":\"保留原始细节的一句话\",\"importance\":1-5,\"occurred_at\":\"可选：事件发生的绝对时间，ISO-8601\"}。\noccurred_at（可选）：这条记忆所述事件**实际发生**的绝对时间，ISO-8601 带时区（如 2026-10-05T09:30:00+08:00）。会话里出现「昨天/前天/上周/上个月」这类相对时间时，按提示末尾给出的当前时间换算成绝对时间再填；没有任何时间依据、或换算不出来时，整条省略该字段——严禁编造，也不要把当前时间当成事件时间填进去。\n若对话涉及编码/调试，可额外提取编码类记忆：\n- rejected_solution：被否决/废弃的实现方案（content 含方案简述 + 被否决原因 + 最终采用方案）\n- pitfall：调试踩坑记录（content 含现象/报错 + 根因 + 解决/规避方法）\n- constraint：项目工程约束（content 含约束描述 + 来源）\n普通闲聊、临时无关对话一律不提取编码类记忆。不要输出任何其他文字。", - "en": "You are the memory curation assistant. From the conversation content below (including user input, assistant replies, tool calls and results), distill atomic memories worth remembering across sessions.\nAtomic-memory principle: each memory holds exactly one standalone fact/preference/decision — short and self-contained (keep raw details like numbers, error messages, commands, paths, and conclusions verbatim in content; do not abstract them away). Prefer splitting into multiple entries over merging and losing detail. Extract 2-4 entries normally, 4-8 for information-dense conversations.\nStance discipline (this wins over the principle above):\n- Record only facts, preferences, decisions, constraints. A preference is the user's stance on a thing — keep recording it. Never record evaluations of the user's own personality, ability, mood or character, favourable or not (\"the user is very smart\", \"the user is frustrated today\" do not belong in memory).\n- The transcript's three roles differ in trustworthiness: \"User:\" is a user statement; \"Assistant:\" and \"Subagent delivery:\" conclusions or evaluations are inferences, not facts. Do not record one as a memory unless the user explicitly adopted it; if it must be kept, say in content that it is an inference. Technical conclusions that tool output, errors or command results state directly are exempt — record those as usual.\n- When a user statement conflicts with an assistant inference, the user statement wins on the facts (never record the assistant inference as fact); but keep both sides as they stand in content — do not write a verdict on who is right and do not adjudicate for the user.\n- importance reflects only how much this fact affects later work. The user's satisfaction, praise or emotion is never a reason to raise it.\nWrite every title and content in English.\nOutput only a JSON array, each item shaped {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"short title\",\"content\":\"one sentence preserving original detail\",\"importance\":1-5,\"occurred_at\":\"optional: absolute time the event happened, ISO-8601\"}.\noccurred_at (optional): the absolute time the described event **actually happened**, in ISO-8601 with a timezone (e.g. 2026-10-05T09:30:00+08:00). When the conversation uses a relative time (yesterday / the day before / last week / last month), convert it into an absolute time against the current time given at the end of this prompt. Omit the field entirely when there is no temporal basis or it cannot be converted — never invent a value, and never pass the current time off as the event time.\nIf the conversation involves coding/debugging, additionally extract coding-specific memories:\n- rejected_solution: implementation approaches that were rejected or abandoned (content: brief approach + why rejected + what was adopted instead)\n- pitfall: debugging pitfalls (content: symptom/error + root cause + fix or workaround)\n- constraint: project engineering constraints (content: constraint description + source)\nNever extract coding-specific memories from casual small talk or temporary unrelated conversations. Output no other text." + "zh": "你是记忆库提炼助手。根据下面的会话内容（含用户输入、助手回答、工具调用与结果），提炼值得跨会话记住的原子记忆。\n原子记忆原则：每条记忆只装一个独立事实/偏好/决策，短小、自带完整上下文（把数字、报错信息、命令、路径、结论等原始细节保留在 content 里，不要抽象概括）；宁可拆成多条也绝不合并丢细节。信息量一般提 2-4 条，信息密集的对话可提 4-8 条。\n立场纪律（与上条原则冲突时以本条为准）：\n- 只记事实、偏好、决策与约束。偏好是用户对某件事的立场，照常记录；但不要记录对用户本人性格、能力、情绪、品行的评价，无论褒贬（「用户很聪明」「用户今天很沮丧」都不进记忆库）。\n- 转录里「用户：」「助手：」「子会话交付：」可信度不同：用户说的话是用户陈述；助手与子会话的结论、评价是推断，不是事实。除非用户明确采纳，否则不要把助手的结论或评价写成记忆；确需保留时，content 里要写明这是推断。工具输出、报错、命令结果直接给出的技术结论不受此限，照常记录。\n- 用户陈述与助手推断冲突时，事实归属以用户陈述为准（不要把助手推断记成事实）；但 content 里要把两种说法如实并存，不要写成谁对谁错、也不要替用户裁决。\n- importance 只看这条事实对后续工作的影响面。用户表达满意、赞许或情绪，都不是提高 importance 的理由。\n只输出 JSON 数组，每项形如 {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"简短标题\",\"content\":\"保留原始细节的一句话\",\"importance\":1-5,\"occurred_at\":\"可选：事件发生的绝对时间，ISO-8601\"}。\noccurred_at（可选）：这条记忆所述事件**实际发生**的绝对时间，ISO-8601 带时区（如 2026-10-05T09:30:00+08:00）。会话里出现「昨天/前天/上周/上个月」这类相对时间时，按提示末尾给出的当前时间换算成绝对时间再填；没有任何时间依据、或换算不出来时，整条省略该字段——严禁编造，也不要把当前时间当成事件时间填进去。\n若对话涉及编码/调试，可额外提取编码类记忆：\n- rejected_solution：被否决/废弃的实现方案（content 含方案简述 + 被否决原因 + 最终采用方案）\n- pitfall：调试踩坑记录（content 含现象/报错 + 根因 + 解决/规避方法）\n- constraint：项目工程约束（content 含约束描述 + 来源）\n同一结论只记一条：若已在 pitfall 写了现象/报错 + 根因 + 规避方法，就不要再单开一条同义 constraint（反之亦然）；时间序列或状态快照类内容按时间顺序写进同一条，不要拆成两条同义条目。\n普通闲聊、临时无关对话一律不提取编码类记忆。不要输出任何其他文字。", + "en": "You are the memory curation assistant. From the conversation content below (including user input, assistant replies, tool calls and results), distill atomic memories worth remembering across sessions.\nAtomic-memory principle: each memory holds exactly one standalone fact/preference/decision — short and self-contained (keep raw details like numbers, error messages, commands, paths, and conclusions verbatim in content; do not abstract them away). Prefer splitting into multiple entries over merging and losing detail. Extract 2-4 entries normally, 4-8 for information-dense conversations.\nStance discipline (this wins over the principle above):\n- Record only facts, preferences, decisions, constraints. A preference is the user's stance on a thing — keep recording it. Never record evaluations of the user's own personality, ability, mood or character, favourable or not (\"the user is very smart\", \"the user is frustrated today\" do not belong in memory).\n- The transcript's three roles differ in trustworthiness: \"User:\" is a user statement; \"Assistant:\" and \"Subagent delivery:\" conclusions or evaluations are inferences, not facts. Do not record one as a memory unless the user explicitly adopted it; if it must be kept, say in content that it is an inference. Technical conclusions that tool output, errors or command results state directly are exempt — record those as usual.\n- When a user statement conflicts with an assistant inference, the user statement wins on the facts (never record the assistant inference as fact); but keep both sides as they stand in content — do not write a verdict on who is right and do not adjudicate for the user.\n- importance reflects only how much this fact affects later work. The user's satisfaction, praise or emotion is never a reason to raise it.\nWrite every title and content in English.\nOutput only a JSON array, each item shaped {\"type\":\"preference|project|decision|history|rejected_solution|pitfall|constraint\",\"title\":\"short title\",\"content\":\"one sentence preserving original detail\",\"importance\":1-5,\"occurred_at\":\"optional: absolute time the event happened, ISO-8601\"}.\noccurred_at (optional): the absolute time the described event **actually happened**, in ISO-8601 with a timezone (e.g. 2026-10-05T09:30:00+08:00). When the conversation uses a relative time (yesterday / the day before / last week / last month), convert it into an absolute time against the current time given at the end of this prompt. Omit the field entirely when there is no temporal basis or it cannot be converted — never invent a value, and never pass the current time off as the event time.\nIf the conversation involves coding/debugging, additionally extract coding-specific memories:\n- rejected_solution: implementation approaches that were rejected or abandoned (content: brief approach + why rejected + what was adopted instead)\n- pitfall: debugging pitfalls (content: symptom/error + root cause + fix or workaround)\n- constraint: project engineering constraints (content: constraint description + source)\nRecord each conclusion once: if pitfall already carries the symptom/error + root cause + fix, do not open a synonymous constraint (and vice versa); keep a time series or state snapshot in one entry in time order instead of splitting it into two synonymous entries.\nNever extract coding-specific memories from casual small talk or temporary unrelated conversations. Output no other text." }, "dreamSummary": { "zh": "你是记忆库状态叙述助手。根据整理后的记忆，写一段 150-200 字的「当前状态」叙述：现在在做什么（活跃主题）、最近完成了什么或发生了什么变化、接下来明显的走向。只陈述记忆中有依据的事实，不要发明细节；偏好、项目、决策自然融入叙述而不必逐项罗列。叙述纪律：只写记忆里已写明的事实与偏好；不要给用户添上性格、能力、情绪方面的评价，也不要把助手自己的判断写成「用户认为」。某条记忆本身带褒贬口气时，只取它的事实内核，不要把那个口气复述进叙述。这段叙述会作为会话上下文常驻注入。只输出叙述文本，不要其他内容。", diff --git a/dsh-mneme/src/service.js b/dsh-mneme/src/service.js index 35f5ecaf..d0b944e9 100644 --- a/dsh-mneme/src/service.js +++ b/dsh-mneme/src/service.js @@ -3403,7 +3403,7 @@ export function createService({ store, mirror, config, onWrite, logger, document }, saveAttr: (r) => store.saveAttr(r), createEntity: (r) => store.createEntity(r), - findEntityByName: (n) => store.findEntityByName(n), + findEntityByName: (n, t) => store.findEntityByName(n, t), findEntityById: (id) => store.findEntityById(id), getAttrsByMemory: (id) => store.getAttrsByMemory(id), // 记忆详情侧栏：一条记忆关联到的实体（entity_attrs.memory_id 反查，纯读）。 diff --git a/dsh-mneme/src/settings.js b/dsh-mneme/src/settings.js index d6e57bf9..ea1124cf 100644 --- a/dsh-mneme/src/settings.js +++ b/dsh-mneme/src/settings.js @@ -195,6 +195,10 @@ const FEATURE_FLAG_BOOLEANS = [ "reflectionUpdateEnabled", "reflectionFailureTracking", "bm25SearchEnabled", + // #687：批内跨类型去重（默认关 = 现状）。同一窗蒸馏里模型偶尔把一条结论同时写成 + // pitfall 与 constraint（或 history 与 project），类型不同，summarizeDedupeMode 的 + // type+title 去重看不见它们。开后可回面板调阈值；判错时被取代方正文仍在库里。 + "summarizeBatchCrossTypeDedupe", "conflictFreezeEnabled", // Issue #603 第 4 层：自动裁决档（默认关）。相似度极高的同 scope 重复对直接 // 保留新条、归档旧条，不进人工队列；模糊地带照旧留给人。会归档记忆，所以必须 @@ -392,6 +396,8 @@ const FEATURE_FLAG_INT_RANGES = { const FEATURE_FLAG_NUMBER_RANGES = { // Issue #127：vector 去重档的并入阈值。 summarizeDedupeMinSim: [0.5, 0.99], + // #687：批内跨类型去重的相似度阈（与 config.js 的 min/max 成对）。 + summarizeBatchCrossTypeMinSim: [0.5, 0.99], // Issue #125：hybrid 判"高相似"的阈值。 dreamCandidateMinSim: [0.5, 0.99], // Issue #24 块2：单次有效触达的边权抬升幅度（与 config.js 的闭区间对齐）。 diff --git a/dsh-mneme/src/store.js b/dsh-mneme/src/store.js index 4b72c5f5..b1f77992 100644 --- a/dsh-mneme/src/store.js +++ b/dsh-mneme/src/store.js @@ -540,11 +540,16 @@ CREATE TABLE IF NOT EXISTS entities ( type TEXT, first_seen TEXT NOT NULL, last_seen TEXT NOT NULL, - mention_count INTEGER DEFAULT 1, - canonical_memory_id TEXT + mention_count INTEGER DEFAULT 1 ); CREATE INDEX IF NOT EXISTS idx_entities_name ON entities(name); CREATE INDEX IF NOT EXISTS idx_entities_type ON entities(type); +-- #688: 写入侧归一的查找键是 lower(trim(name)) + type（见 findEntityByName）。表达式索引 +-- 让这条查询不必扫全表——本机实体表 25,474 行，实体抽取是每条记忆都要走的写入热路径。 +-- 折叠在 SQL 里做（不是先在 JS 里折好再绑），是为了让「命中键」与「索引键」逐字节同源： +-- JS 的 toLowerCase 与 SQLite 的 lower() 只在 ASCII 上一致，混用会在含特殊字符的名字上 +-- 造出一行查不到自己的孤儿。存量行不改写，索引只是给旧值补一个可查的键。 +CREATE INDEX IF NOT EXISTS idx_entities_name_key ON entities(lower(trim(name)), type); CREATE TABLE IF NOT EXISTS entity_attrs ( id TEXT PRIMARY KEY, @@ -1215,8 +1220,7 @@ function toEntity(row) { type: row.type ?? undefined, first_seen: row.first_seen, last_seen: row.last_seen, - mention_count: row.mention_count, - canonical_memory_id: row.canonical_memory_id ?? undefined + mention_count: row.mention_count }; } @@ -1479,6 +1483,29 @@ export function createStore(path) { // 行缺列 → 默认 manual 1.0（最保守：既有语义不因迁移而变弱）。 addColumn("entity_relations", "weight", "ALTER TABLE entity_relations ADD COLUMN weight REAL NOT NULL DEFAULT 1.0"); addColumn("entity_relations", "source", "ALTER TABLE entity_relations ADD COLUMN source TEXT NOT NULL DEFAULT 'manual'"); + + // #688：entities.canonical_memory_id 是 v0.7.1 预留的「规范行指针」，此后长成了 + // 一根孤儿——全仓只有 plumbing（DDL、行映射、INSERT、patch、UPDATE），没有一处 SELECT + // 或业务写入，真实库里 25,474 行全是 NULL。留着它不是中性的：下一位读者会把它当成 + // 一条可用的判据（#688 的实体归并讨论里就差点被当成「同一条记忆的实体该并」）。 + // 删列没有数据要保（本来就全空），也不需要 applied_migrations 登记——判定条件是 + // 「列还在不在」，跑完恒不命中，和 addColumn 同一族。 + // + // 前提：SQLite >= 3.35 才有 DROP COLUMN（本项目 node:sqlite 在支持的运行时上是 + // 3.46+，本机 3.51.3）。竞态与 addColumn 同理：两个进程可能同时通过检查，后到的 + // 那个收到的是 "no such column"，那是预期结局而不是错误。 + const dropColumn = (table, column) => { + const cols = db.prepare(`PRAGMA table_info(${table})`).all().map((c) => c.name); + if (!cols.includes(column)) return false; + try { + db.exec(`ALTER TABLE ${table} DROP COLUMN ${column}`); + return true; + } catch (e) { + if (/no such column/i.test(String(e?.message ?? e))) return false; + throw e; + } + }; + dropColumn("entities", "canonical_memory_id"); // 列序是 content_hash 打头：哈希几乎唯一，等值 seek 就已收窄到候选行，type 只当同 // 一次 seek 里的第二列过滤；反过来（type 打头）下面的存量回填就得全表扫——每次打开 // 都要把全部正文读一遍。 @@ -4685,15 +4712,49 @@ export function createStore(path) { function createEntity({ name, type }) { const id = randomUUID(); const now = nowIso(); + // 落库前只去掉首尾空白，保留原大小写：显示名是给人看的（"DeepSeek" 不该被写成 + // "deepseek"），大小写折叠只发生在查找键上。存量行按 #688 口径一行不动。 + const storedName = String(name ?? "").trim(); db.prepare( - `INSERT INTO entities (id, name, type, first_seen, last_seen, mention_count, canonical_memory_id) - VALUES (?, ?, ?, ?, ?, ?, ?)` - ).run(id, name, type ?? null, now, now, 1, null); + `INSERT INTO entities (id, name, type, first_seen, last_seen, mention_count) + VALUES (?, ?, ?, ?, ?, ?)` + ).run(id, storedName, type ?? null, now, now, 1); return toEntity(db.prepare("SELECT * FROM entities WHERE id = ?").get(id)); } - function findEntityByName(name) { - return toEntity(db.prepare("SELECT * FROM entities WHERE name = ?").get(name)); + /** + * Resolve an entity by its normalized key: lower(trim(name)), optionally + * narrowed to one type. + * + * #688: the pre-fix lookup was `WHERE name = ?` — byte equality. "DeepSeek" and + * "deepseek " were two rows forever, even though the extractor means the same + * object; the measured stock was 170 same-type groups collapsed by this fold. + * + * Why the type dimension is not optional in the write path: folding name alone + * merges 95 groups that are genuinely different objects — "DeepSeek" the + * technology vs "deepseek" the organization is the canonical example. Callers + * that resolve a *mention* must pass the type; callers that look up by a + * user-supplied name (panel search) may omit it and get the most recently seen + * row across types. + * + * Why no version-suffix stripping: "DeepSeek-V3" and "DeepSeek-V4" are distinct + * models; collapsing them loses which one a constraint is about. + * + * @param {string} name + * @param {string|null} [type] entity type; when undefined/null the lookup is + * type-agnostic (panel / entity-name search path) + * @returns {object|undefined} + */ + function findEntityByName(name, type) { + if (name === undefined || name === null) return undefined; + const hasType = type !== undefined && type !== null; + const sql = hasType + ? `SELECT * FROM entities WHERE lower(trim(name)) = lower(trim(?)) AND type IS ? + ORDER BY last_seen DESC LIMIT 1` + : `SELECT * FROM entities WHERE lower(trim(name)) = lower(trim(?)) + ORDER BY last_seen DESC LIMIT 1`; + const params = hasType ? [String(name), String(type)] : [String(name)]; + return toEntity(db.prepare(sql).get(...params)); } function findEntityById(id) { @@ -4711,16 +4772,13 @@ export function createStore(path) { const has = (k) => Object.prototype.hasOwnProperty.call(patch, k); const name = has("name") ? patch.name : old.name; const type = has("type") ? patch.type : old.type; - const canonical_memory_id = has("canonical_memory_id") - ? patch.canonical_memory_id - : old.canonical_memory_id; const mention_count = has("mention_count") ? patch.mention_count : (old.mention_count ?? 1) + 1; const now = nowIso(); db.prepare( - `UPDATE entities SET name = ?, type = ?, last_seen = ?, mention_count = ?, canonical_memory_id = ? WHERE id = ?` - ).run(name, type ?? null, now, mention_count, canonical_memory_id ?? null, id); + `UPDATE entities SET name = ?, type = ?, last_seen = ?, mention_count = ? WHERE id = ?` + ).run(name, type ?? null, now, mention_count, id); return findEntityById(id); } diff --git a/dsh-mneme/src/summarize.js b/dsh-mneme/src/summarize.js index 7e4f1d73..dbb1aab0 100644 --- a/dsh-mneme/src/summarize.js +++ b/dsh-mneme/src/summarize.js @@ -17,6 +17,10 @@ import { contentToText, countImages, IMAGE_PLACEHOLDER_PREFIX } from "./attachme // 图像描述档（docs/DISTILL.md:346）：占位之外再补一段「图里是什么」。本文件只 // 负责把描述并进转录行；出描述的调用、准入判断都在 image-describe.js。 import { appendDescriptions, describeImages, imageBlocks, canSeeImages, DESCRIPTION_PREFIX, imageRefOf } from "./image-describe.js"; +// #687：批内跨类型去重沿用写入侧质量过滤那把相似度（同一份归一化 bigram 重叠、 +// 同一族阈值），不另造第二套度量——两套度量会给出两套「像不像」的答案，而这一条 +// 判据只该有一个。 +import { textSimilarity } from "./quality-filter.js"; // #512：每轮蒸馏一个 run id（归因主体）。store 侧也能铸造，但这里必须知道它—— // 写记忆时要把它一路带到 saveMemoryProvenance / 落 run 行，拿回来再写就是两次往返。 import { randomUUID } from "node:crypto"; @@ -38,6 +42,87 @@ const OCCURRED_AT_FUTURE_TOLERANCE_MS = 24 * 60 * 60 * 1000; // 刻意不做成配置键：护栏本身默认关（summarizeParseFailureMaxRetries=0），一个只为 // 护栏服务的失效窗口不该再占一档用户可见配置（AGENTS.md 旗标计数锁也要跟着动）。 const PARSE_FAILURE_STREAK_STALE_MS = 7 * 24 * 60 * 60 * 1000; + +// --- #687 批内跨类型去重 --------------------------------------------------------- +// 一次蒸馏里，同一个结论常被写成两条不同类型的记忆（"踩坑" 与 "约束" 其实是同一句 +// 话；一次事件与它所属的项目状态写了两遍）。2026-11-13 在真实库上量过：跨类型余弦 +// >= 0.90 的 45 对里 36 对（80%）同义，集中在这两组类型对上（constraint<->pitfall 9、 +// history<->project 8）。但同一批数据里还有 9 对（20%）合并会丢信息：5 对是时间序列 +// /状态快照、2 对是「症状」与「修法」、1 对结论相反、1 对版本号不同 —— 所以不开 +// 常开的类型闸，只在**同一批**（同一窗蒸馏、同一件事）上认这两组。 +export const BATCH_CROSS_TYPE_PAIRS = Object.freeze([ + Object.freeze(["pitfall", "constraint"]), + Object.freeze(["history", "project"]) +]); +// 与 quality-filter 的 duplicate 信号同一档（0.85）：同一条判据「这两段是不是在说 +// 同一件事」在写入侧已经用这个数，这里再拍一个相近的数只会让两处阈值互相漂移。 +export const BATCH_CROSS_TYPE_MIN_SIM = 0.85; +// 下面两条不是阈值旋钮，是「相似」与「同一结论」之间的分隔线： +// - 太短的条目不参与：textSimilarity 以较小一侧为分母，一句 15 字的话几乎完全被任何 +// 长段落包含 → 相似度接近 1，但那是「被提到」而不是「重复」； +// - 长度悬殊的条目也不参与（短边不到长边一半）：那是包含关系（一段长记录里顺手提了 +// 另一个结论），合并它正是在丢信息。 +const BATCH_CROSS_TYPE_MIN_CHARS = 24; +const BATCH_CROSS_TYPE_MIN_LEN_RATIO = 0.5; + +function batchPairKey(a, b) { + for (const [left, right] of BATCH_CROSS_TYPE_PAIRS) { + if ((a === left && b === right) || (a === right && b === left)) return `${left}<->${right}`; + } + return null; +} + +/** + * Plan the in-batch cross-type supersede pass over one distillation window's + * entries. Pure: no I/O, no ids — it returns indices, because the rows do not + * exist yet (the caller chains them after writing). + * + * Pairs are visited by descending similarity so the strongest evidence wins the + * right to retire an entry; an entry is retired at most once and may not retire + * another entry that is itself retired (no chains that would hide a row behind a + * row that is already hidden). + * + * Winner = the entry carrying more content; equal length falls back to the entry + * written earlier in the batch (lower index). "More content" rather than "newer" + * because both were produced by the same call — there is no recency to speak of, + * and the fuller text is what a supersede should preserve. + * + * @param {Array<{type?:string,title?:string,content?:string}>} entries + * @param {{minSim?:number}} [options] + * @returns {Array<{winnerIndex:number, loserIndex:number, sim:number, pair:string}>} + */ +export function planBatchCrossTypeDedupe(entries, { minSim = BATCH_CROSS_TYPE_MIN_SIM } = {}) { + const plans = []; + if (!Array.isArray(entries) || entries.length < 2) return plans; + const candidates = []; + for (let i = 0; i < entries.length; i++) { + for (let j = i + 1; j < entries.length; j++) { + const a = entries[i] ?? {}; + const b = entries[j] ?? {}; + const pair = batchPairKey(a.type, b.type); + if (!pair) continue; + const la = String(a.content ?? "").length; + const lb = String(b.content ?? "").length; + const shortSide = Math.min(la, lb); + const longSide = Math.max(la, lb); + if (shortSide < BATCH_CROSS_TYPE_MIN_CHARS) continue; + if (shortSide / longSide < BATCH_CROSS_TYPE_MIN_LEN_RATIO) continue; + const sim = textSimilarity(`${a.title ?? ""}\n${a.content ?? ""}`, `${b.title ?? ""}\n${b.content ?? ""}`); + if (sim < minSim) continue; + const winnerIndex = la >= lb ? i : j; + const loserIndex = winnerIndex === i ? j : i; + candidates.push({ winnerIndex, loserIndex, sim, pair }); + } + } + candidates.sort((x, y) => y.sim - x.sim || x.loserIndex - y.loserIndex); + const retired = new Set(); + for (const c of candidates) { + if (retired.has(c.loserIndex) || retired.has(c.winnerIndex)) continue; + retired.add(c.loserIndex); + plans.push({ winnerIndex: c.winnerIndex, loserIndex: c.loserIndex, sim: Number(c.sim.toFixed(4)), pair: c.pair }); + } + return plans; +} function normalizeDistilledOccurredAt(raw, nowMs = Date.now()) { if (typeof raw !== "string") return undefined; const text = raw.trim(); @@ -1233,6 +1318,10 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = const dedupeMode = config.summarizeDedupeMode ?? "off"; let deduped = 0; let dedupeMaxSim = 0; + // #687：批内跨类型去重的落账计数（链成功几对）。默认关着一律为 0， + // 审计行与加它之前逐字节一致。 + let batchDeduped = 0; + const batchChainedPairs = []; // v0.9.0（M.O. 项目记忆）：一次窗口解析一次——同一蒸馏窗口里所有条目同属一个 // 工作区、一个 agent 身份，逐条解析是白花钱。解析不到就落 NULL（NULL = 未标注 = // 全局可见），且解析器绝不抛错（scope.js 的硬约束：记忆写入不该被作用域拖挂）。 @@ -1269,6 +1358,31 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = } writes.push({ entry, source, dup }); } + // #687：批内跨类型去重（opt-in，默认关 = 现状）。同一窗蒸馏里模型常把一个结论 + // 写成两条不同类型（pitfall 与 constraint 同义、history 与 project 同义）， + // 而"类型不同"恰好让写入侧的 type+title 去重看不见它们。合并方向固定为 + // supersede：正文零改写、不 merge —— 判错时原文仍在库里，只是默认注入不再取它， + // 而 merge 会真的抹掉一条的措辞，没有回头路。 + // + // 已被并进既有行的条目（dup 有值）不参与：它的正文已经进了那条老记忆，把老记忆 + // 盖到下一条新条目下面，等于用一次蒸馏顺手隐藏一条累积了很久的旧行。 + const batchPlans = []; + if (config?.summarizeBatchCrossTypeDedupe !== false && writes.length > 1) { + const plannable = writes + .map((write, index) => ({ entry: write.entry, index })) + .filter((slot) => !writes[slot.index].dup); + const planned = planBatchCrossTypeDedupe(plannable.map((slot) => slot.entry), { + minSim: config?.summarizeBatchCrossTypeMinSim ?? BATCH_CROSS_TYPE_MIN_SIM + }); + for (const plan of planned) { + batchPlans.push({ + winnerIndex: plannable[plan.winnerIndex].index, + loserIndex: plannable[plan.loserIndex].index, + sim: plan.sim, + pair: plan.pair + }); + } + } // #399：蒸馏窗口的原始公开记录 + 记忆级溯源链接，与记忆写入同事务落盘。 // 凭证与它的派生记忆必须同生共死：记忆写了而记录没写，溯源就断在「不知道 // 出自哪段原文」；记录写了而记忆没写，盘上留一份没有索引的原文。默认关 @@ -1280,7 +1394,9 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = : null; const writeWindow = () => { const savedIds = []; - for (const { entry, source, dup } of writes) { + // #687：批内 supersede 需要「写之后」才有的 id，按 writes 的下标记住它们。 + const idByWriteIndex = new Map(); + for (const [writeIndex, { entry, source, dup }] of writes.entries()) { const outcome = service.saveWithDedupe({ ...entry, source, @@ -1301,6 +1417,19 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = // created 与 merged 两条成功分支都返回 memory（含 id）；merged 时 id 是 // 被并入的既有记忆——本次窗口的原文确实贡献了它，链接它是诚实的。 if (outcome?.memory?.id) savedIds.push(outcome.memory.id); + if (outcome?.memory?.id) idByWriteIndex.set(writeIndex, outcome.memory.id); + } + // #687：两条都落库之后才链——supersedeChain 写的是行间指针，先链后写会指向 + // 一个还不存在的 id。链失败（行被并发归档/物理删除）就跳过：这一对的正文都 + // 已经写进去了，宁可留着重复，也不能为了「看起来干净」去伪造一次链。 + for (const plan of batchPlans) { + const loserId = idByWriteIndex.get(plan.loserIndex); + const winnerId = idByWriteIndex.get(plan.winnerIndex); + if (!loserId || !winnerId || loserId === winnerId) continue; + if (service.supersedeChain?.(loserId, winnerId)) { + batchDeduped++; + batchChainedPairs.push(plan.pair); + } } // #512：run 的条数口径是「本次写成功的记忆行数」，与 savedIds 同源—— // 另起一个计数器会在 merged 分支上与它对不上（那一路也 push 了 id）。 @@ -1401,7 +1530,11 @@ export function createSummarizer(ctx, service, config, deps = {}, sessionGate = parsed: parsed.length, ...(parsedResult.salvaged ? { json_salvaged: true } : {}), ...(capped > 0 ? { capped } : {}), - ...(deduped > 0 ? { deduped, mode: dedupeMode, maxSim: Number(dedupeMaxSim.toFixed(4)) } : {}) + ...(deduped > 0 ? { deduped, mode: dedupeMode, maxSim: Number(dedupeMaxSim.toFixed(4)) } : {}), + // #687：批内跨类型 supersede 的落账（几对、为什么链）。默认关时这个键根本不出现。 + ...(batchDeduped > 0 + ? { batchDeduped, batchPairs: batchChainedPairs } + : {}) }; } // 记忆写入和解析都成功后才提交窗口；流失败、中止、解析失败或写入异常 diff --git a/dsh-mneme/test/api.test.js b/dsh-mneme/test/api.test.js index 77482c34..de878cd7 100644 --- a/dsh-mneme/test/api.test.js +++ b/dsh-mneme/test/api.test.js @@ -875,7 +875,10 @@ test("GET /api/dsh-mneme/features returns empty overrides and effective config d // = +1，故 134 + 1 = 135。 // 图像溯源（v2）新增 imageMemoryEnabled（顶层布尔键、imageDescribeEnabled 的子 // 开关，走 BOOLEANS 表）= +1，故 135 + 1 = 136。 - assert.equal(Object.keys(data.effective).length, 136); + // #687 批内跨类型去重新增两个键：summarizeBatchCrossTypeDedupe（顶层布尔键， + // 走 BOOLEANS 表）与 summarizeBatchCrossTypeMinSim（相似度阈，走 + // NUMBER_RANGES 表）= +2，故 136 + 2 = 138。 + assert.equal(Object.keys(data.effective).length, 138); // 描述档三条的默认值单独钉：开关默认关（不配路由就一个调用都不发）， // 路由默认空串——空串在这里的语义是「整档不跑」，不是「跟随当前模型」， // 所以它不能像 summarizeProvider 那样被回填成会话模型。 diff --git a/dsh-mneme/test/entities.test.js b/dsh-mneme/test/entities.test.js index 56e9627b..cd25679b 100644 --- a/dsh-mneme/test/entities.test.js +++ b/dsh-mneme/test/entities.test.js @@ -709,3 +709,78 @@ test("#372: no unbounded cap — large transaction drains every entry", () => { }); assert.equal(calls, 150, "no silent drop at any queue cap"); }); + +// ============================================================ #688 写入侧归一 + +// 归一的口径只有一条：查找键 = lower(trim(name)) + type。折叠不能预做在 JS 里 +// 再绑参——JS 的 toLowerCase 与 SQLite 的 lower() 只在 ASCII 上一致，混用会在 +// 特殊字符上造出「一行查不到自己」的孤儿（见 store.js 的索引注释）。 +test("#688 findEntityByName folds case/whitespace and keeps the type dimension", () => { + const store = openStore(); + const tech = store.createEntity({ name: "DeepSeek", type: "technology" }); + const org = store.createEntity({ name: "deepseek", type: "organization" }); + + // 不带 type：仍能命中（面板/名字检索走这条），名字折叠生效。 + assert.ok(store.findEntityByName(" deepseek "), "case + surrounding whitespace folded"); + + // 带 type：类型必须一起进键。这 95 组跨类型同名正是 #688 不能裸折叠的理由—— + // 只按名字归一会把技术 DeepSeek 与组织 deepseek 并成一个。 + assert.equal(store.findEntityByName("deepseek", "technology").id, tech.id); + assert.equal(store.findEntityByName(" DEEPSEEK ", "organization").id, org.id); + assert.equal(store.findEntityByName("deepseek", "person"), undefined, "type still filters"); + + // 版本号后缀不参与归一：DeepSeek-V3 与 V4 是两个模型，剥后缀就并错了。 + const v3 = store.createEntity({ name: "DeepSeek-V3", type: "technology" }); + assert.equal(store.findEntityByName("deepseek-v4", "technology"), undefined, "no version-suffix stripping"); + assert.equal(store.findEntityByName("deepseek-v3", "technology").id, v3.id); + store.close(); +}); + +test("#688 createEntity trims the stored name but preserves its case", () => { + const store = openStore(); + const ent = store.createEntity({ name: " Vite ", type: "technology" }); + assert.equal(ent.name, "Vite", "surrounding whitespace is not stored"); + assert.equal(store.findEntityByName("vite", "technology").id, ent.id); + store.close(); +}); + +// 迁移：老库的 entities 带 canonical_memory_id（v0.7.1 的预留列，全仓无读取方）。 +// 打开即删列；数据与其余列原样保留；二次打开不抛（判定条件是「列还在不在」）。 +test("#688 legacy entities.canonical_memory_id is dropped on open, idempotently", () => { + const dir = mkdtempSync(join(tmpdir(), "dsh-mneme-entity-drop-")); + const path = join(dir, "legacy-drop.db"); + try { + const legacy = new DatabaseSync(path); + legacy.exec(`CREATE TABLE memories ( + id TEXT PRIMARY KEY, type TEXT NOT NULL, title TEXT NOT NULL, content TEXT NOT NULL, + tags TEXT NOT NULL DEFAULT '[]', importance INTEGER NOT NULL DEFAULT 3, + forgotten INTEGER NOT NULL DEFAULT 0, source TEXT, created_at TEXT NOT NULL, updated_at TEXT NOT NULL + );`); + legacy.exec(`CREATE TABLE entities ( + id TEXT PRIMARY KEY, name TEXT NOT NULL, type TEXT, + first_seen TEXT NOT NULL, last_seen TEXT NOT NULL, + mention_count INTEGER DEFAULT 1, canonical_memory_id TEXT + );`); + legacy.prepare("INSERT INTO entities (id,name,type,first_seen,last_seen,mention_count,canonical_memory_id) VALUES (?,?,?,?,?,?,?)") + .run("e1", "遗留实体", "concept", "2026-01-01T00:00:00.000Z", "2026-01-01T00:00:00.000Z", 3, "mem-1"); + legacy.close(); + + const store = createStore(path); + const cols = store.db.prepare("PRAGMA table_info(entities)").all().map((c) => c.name); + assert.ok(!cols.includes("canonical_memory_id"), "the orphan column is gone after open"); + const row = store.findEntityById("e1"); + assert.equal(row.name, "遗留实体", "existing row survives the drop"); + assert.equal(row.mention_count, 3, "other columns keep their values"); + assert.equal(row.canonical_memory_id, undefined, "row mapping no longer exposes it"); + store.close(); + + // 幂等：第二次打开列已不在，不能让 PRAGMA 检查之外的路径抛错。 + const again = createStore(path); + assert.ok(!again.db.prepare("PRAGMA table_info(entities)").all().map((c) => c.name).includes("canonical_memory_id")); + again.createEntity({ name: "新实体", type: "concept" }); + assert.ok(again.findEntityByName("新实体", "concept"), "reopen stays writable"); + again.close(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/dsh-mneme/test/summarize.test.js b/dsh-mneme/test/summarize.test.js index af1ed60e..ff9b82db 100644 --- a/dsh-mneme/test/summarize.test.js +++ b/dsh-mneme/test/summarize.test.js @@ -8,7 +8,7 @@ import { Context } from "@deepseek-ai/cordis"; import * as mneme from "../src/index.js"; import { createStore } from "../src/store.js"; import { createService } from "../src/service.js"; -import { createSummarizer, parseSummaryJson, parsePeakSpec, isInPeakWindow, nextOffPeakAt } from "../src/summarize.js"; +import { createSummarizer, parseSummaryJson, parsePeakSpec, isInPeakWindow, nextOffPeakAt, planBatchCrossTypeDedupe } from "../src/summarize.js"; import { createSettings } from "../src/settings.js"; // Issue #239 第 4 项：可注入的假时钟 + 定时器——错峰队列的测试必须能「推进时间」， @@ -2249,3 +2249,125 @@ test("#493 补充：停了很久的会话，旧 streak 失效，不会被几周 summarizer.dispose(); store.close(); }); + +// --- #687 批内跨类型去重（opt-in，默认关） ---------------------------------- +// 模型会把一个结论写成两条不同类型（pitfall 与 constraint 同义 / history 与 +// project 同义），类型不同正好让 summarizeDedupeMode 的 type+title 去重看不见。 +// 处置固定为 supersede：正文零改写、不 merge——判错时原文仍在库里，只是默认注入 +// 不再取它；merge 会真的抹掉一条的措辞，没有回头路。 + +const BATCH_DUP_PITFALL = "迁移实体表时忘了先确认 SQLite 版本，DROP COLUMN 在旧版本上会直接报错，只能重建整张表再逐条重指外键，恢复花了两小时。"; +const BATCH_DUP_CONSTRAINT = "迁移实体表前先确认 SQLite 版本：DROP COLUMN 在旧版本上会直接报错，只能重建整张表再逐条重指外键。"; + +test("#687 planBatchCrossTypeDedupe only pairs the two allowed type groups", () => { + const entries = [ + { type: "pitfall", title: "迁移实体表", content: BATCH_DUP_PITFALL }, + { type: "constraint", title: "迁移实体表", content: BATCH_DUP_CONSTRAINT }, + // 同类型对（decision↔decision）与未登记的跨类型对（decision↔pitfall）都不许合并： + // 45 对全跨类型里有 20% 合并会丢信息，类型闸只在两组上常开是刻意的。 + { type: "decision", title: "迁移实体表", content: BATCH_DUP_PITFALL }, + { type: "decision", title: "迁移实体表", content: BATCH_DUP_CONSTRAINT } + ]; + const plans = planBatchCrossTypeDedupe(entries); + assert.equal(plans.length, 1, "only the pitfall↔constraint pair is eligible"); + assert.equal(plans[0].pair, "pitfall<->constraint"); + assert.equal(plans[0].winnerIndex, 0, "the fuller text wins"); + assert.equal(plans[0].loserIndex, 1); + assert.ok(plans[0].sim >= 0.85, `near-identical twins must clear the bar (got ${plans[0].sim})`); +}); + +test("#687 planBatchCrossTypeDedupe skips too-short and lopsided entries", () => { + // textSimilarity 以较小一侧为分母：一句短话几乎被任何长段落包含 → 相似度接近 1。 + // 那是「被提到」而不是「重复」，所以短于 24 字的一侧不参与。 + assert.deepEqual(planBatchCrossTypeDedupe([ + { type: "pitfall", title: "报错", content: "报错了" }, + { type: "constraint", title: "报错", content: "报错了" } + ]), [], "a 3-char twin must not be treated as a duplicate conclusion"); + + // 长度悬殊（短边不到长边一半）是包含关系，不是重复。 + const lopsided = "迁移实体表时先确认 SQLite 版本，DROP COLUMN 需要 3.35 以上。" + "这条记录里还顺手写了一大段与本结论无关的现场环境、会话编号与复现步骤，长度远超一半。".repeat(3); + assert.deepEqual(planBatchCrossTypeDedupe([ + { type: "pitfall", title: "迁移", content: lopsided }, + { type: "constraint", title: "迁移", content: "迁移实体表时先确认 SQLite 版本，DROP COLUMN 需要 3.35 以上。" } + ]), [], "a short note inside a long record is containment, not duplication"); +}); + +test("#687 planBatchCrossTypeDedupe retires an entry at most once", () => { + // 一条短的 constraint 同时与两条 pitfall 高度相似：只能被取代一次，且不能 + // 先把 A 藏到 B 后面、再把 B 藏到 C 后面（那会造出两层看不见的链）。 + const plans = planBatchCrossTypeDedupe([ + { type: "pitfall", title: "一", content: BATCH_DUP_PITFALL }, + { type: "constraint", title: "二", content: BATCH_DUP_CONSTRAINT }, + { type: "pitfall", title: "三", content: BATCH_DUP_PITFALL + "复查时补了一条同样的教训。" } + ]); + assert.equal(plans.length, 1, "the shared short entry may only be retired once"); + assert.equal(plans[0].loserIndex, 1); + assert.ok(!plans.some((p) => p.winnerIndex === 1), "a retired entry may never host another retire"); +}); + +test("#687 batch dedupe can be switched off: twins are written untouched", async () => { + const stream = () => (async function* () { + yield { type: "block-start", block: { type: "text" } }; + yield { type: "text-delta", delta: JSON.stringify([ + { type: "pitfall", title: "迁移实体表", content: BATCH_DUP_PITFALL, importance: 4 }, + { type: "constraint", title: "迁移实体表", content: BATCH_DUP_CONSTRAINT, importance: 4 } + ]) }; + yield { type: "block-end", block: { type: "text" } }; + yield { type: "finish", kind: "ok" }; + })(); + const { events, store, service, summarizer } = setup({ summarizeBatchCrossTypeDedupe: false }, { stream }); + const handler = events.find((e) => e.name === "session/event").fn; + const session = { + id: "s687-off", + requestHeader: () => ({ config: { provider: "deepseek", model: "deepseek-chat" } }), + events: [userMessage("帮我迁移实体表", 1), { seq: 2, type: "turn/end" }] + }; + await handler(session, { seq: 2, type: "turn/end" }); + + assert.equal(store.count(), 2, "kill switch stays byte-identical: nothing is chained"); + for (const m of store.all()) assert.equal(m.superseded_by, undefined, `${m.type} must stay live`); + const ok = service.listLlmAudits().find((r) => r.operation_type === "summarize_compress" && r.status === "success"); + assert.ok(ok, "the summarize audit line exists"); + assert.equal(ok.metadata?.batchDeduped, undefined, "no audit noise when the gate is off"); + summarizer.dispose(); + store.close(); +}); + +test("#687 batch dedupe supersedes the weaker twin without rewriting its text", async () => { + const stream = () => (async function* () { + yield { type: "block-start", block: { type: "text" } }; + yield { type: "text-delta", delta: JSON.stringify([ + { type: "pitfall", title: "迁移实体表", content: BATCH_DUP_PITFALL, importance: 4 }, + { type: "constraint", title: "迁移实体表", content: BATCH_DUP_CONSTRAINT, importance: 4 } + ]) }; + yield { type: "block-end", block: { type: "text" } }; + yield { type: "finish", kind: "ok" }; + })(); + // 不带开关 = 走默认值。2026-11-13 起默认开（#687 的拍板），所以这条同时是默认路径的证明。 + const { events, store, service, summarizer } = setup({}, { stream }); + const handler = events.find((e) => e.name === "session/event").fn; + const session = { + id: "s687-on", + requestHeader: () => ({ config: { provider: "deepseek", model: "deepseek-chat" } }), + events: [userMessage("帮我迁移实体表", 1), { seq: 2, type: "turn/end" }] + }; + await handler(session, { seq: 2, type: "turn/end" }); + + const all = store.all(); + assert.equal(all.length, 2, "supersede never deletes: both rows stay in the library"); + const winner = all.find((m) => m.type === "pitfall"); + const loser = all.find((m) => m.type === "constraint"); + assert.ok(winner && loser); + assert.equal(loser.superseded_by, winner.id, "the shorter twin points at the fuller one"); + assert.ok(loser.superseded_at, "the chain is timestamped"); + assert.equal(loser.content, BATCH_DUP_CONSTRAINT, "loser text must be byte-identical — zero rewrite"); + assert.equal(winner.content, BATCH_DUP_PITFALL); + assert.equal(winner.superseded_by, undefined, "the winner stays live"); + + const ok = service.listLlmAudits().find((r) => r.operation_type === "summarize_compress" && r.status === "success"); + assert.ok(ok, "the summarize audit line exists"); + assert.equal(ok.metadata?.batchDeduped, 1, "the audit line records the disposition"); + assert.deepEqual(ok.metadata?.batchPairs, ["pitfall<->constraint"]); + summarizer.dispose(); + store.close(); +});