Skip to content

Extension namespaces: session_ext + kv_store (schema v4, #3069) - #3070

Merged
ranxianglei merged 6 commits into
masterfrom
2026-10-14_storage-ext
Oct 14, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-10-14_storage-ext

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Closes #3069.

What

Two generic tables in the unified index, so host/plugin/UI features get a durable home WITHOUT per-feature schema migrations (the zero-baggage window before the full-default flip):

  • session_ext(session_id, key) → value_json — per-session arbitrary state. Swept inside the deleteSession transaction (state dies with the session; checkpoint it if it must survive). No pin semantics.
  • kv_store(namespace, key) → value_json — namespaced cross-session state (lane-level / per-agent / global). Caller-managed lifecycle: deleteKv("lane:<id>") retires a lane namespace; deleteSession never touches it.

Design edges (deliberate)

  • Kernel state never migrates here — kstate_json stays the kernel's. Both tables sit OUTSIDE the LegacySessionLike view → zero digest parity churn (corpus + goldens prove it).
  • Keys/namespaces: non-empty printable ≤256 chars, namespaced by convention ("r3:"", "web:", "plugin::", "lane::"`). Guards reject control chars, oversize and non-serializable values.
  • Values: inline JSON ≤4 MiB (large payloads belong in the content CAS — different shape: small + frequently rewritten beats dedup).
  • listKv key-prefix filters escape LIKE metacharacters (%/_/\\ are literal).
  • Available in shadow AND full (side-table substrate like checkpoints — not part of the load path); legacy-only fails fast with the same 501-able message the replica routes use.

Surface

UnifiedStore: set/get/list/deleteSessionExt, set/get/list/deleteKv. UnifiedShadow: passthroughs. SessionStore: thin liveShadow() wrappers.

Migration

v3 → v4 = two idempotent CREATE TABLE IF NOT EXISTS inside the existing migration transaction. Fresh DBs create v4 directly. Tests: round-trips/upsert/prefix isolation, sweep-vs-kv survival on deleteSession, guards, v3→v4 in-place + idempotent reopen, legacy 501 class, schema assertions 3→4.

In passing: golden regen REVERTED (phantom drift, my stale kernel/dist)

I first believed the 4 storage-render goldens were red on master and regen'd them (bf32bbb). Wrong: my local kernel/dist predated #3058's preserveRecentMessages 5→3 flip, so my drive produced the pre-flip digests. After cd kernel && npm run build, master's committed goldens (8f93c33) pass byte-for-byte — reverted in daebe97. tsc does NOT catch a stale kernel dist (no new exports involved); the only guard is building kernel before regen. Suggested hardening (in #3075): freshness assertion inside update-storage-render-goldens.ts.

Known master-side red (NOT from this PR): #1387 stop-signal tests

Plain origin/master (c800ed1, fresh kernel build) fails the two #1387 tail tests — the #3058 preserve 5→3 flip changed drain/partial-failure stop-signal semantics the tests pin. Filed #3075 for the owner to decide (test update vs kernel floor). The test-matrix legs failing on this PR are exactly those two; every other check including typecheck/lint/goldens is green.

Tests

  • tests/storage-ext.test.ts — 5/5
  • storage family regression: unified 13, corpus 9, replicas 4, ccr 3, review-fixes 4, identity 6, engine-proofs 12, json-writer-off 3, fork-convergence 2, persist-tail 7, export-jsonl 4 — all green
  • goldens 5/5 (master's committed values, fresh kernel); tsc clean; unused-export gate clean (2 waived)

prepared by an agent under the local git identity (ranxianglei), human merge required

…v4) (#3056)

Two generic tables give host/plugin/UI features a durable home in the
unified index without per-feature schema migrations:

- session_ext (session_id, key) → value_json: per-session arbitrary
  state, swept by deleteSession (no pin semantics — checkpoint instead);
- kv_store (namespace, key) → value_json: namespaced cross-session
  state (lane-level / per-agent / global), caller-managed lifecycle
  (deleteKv("lane:<id>") retires a lane namespace).

Keys/namespaces: non-empty printable ≤256 chars (namespaced by
convention: "r3:", "web:", "plugin:<name>:", "lane:<id>:").
Values: inline JSON ≤4 MiB (large payloads belong in the content CAS).
LIKE metacharacters escaped in listKv key-prefix filters.

Kernel state NEVER migrates here — kstate_json stays the kernel's and
the tables are outside the LegacySessionLike view (zero digest parity
churn). Available in shadow AND full (side-table substrate like
checkpoints); legacy-only fails fast with the 501-able message.

Store: set/get/list/delete for both tables; deleteSession sweeps
session_ext inside its transaction. SessionStore: thin liveShadow()
wrappers. Migration v3→v4 is two idempotent CREATE TABLE IF NOT EXISTS.

Tests: round-trips/upsert/prefix-isolation, sweep-vs-kv survival,
guards, v3→v4 in-place migration + idempotence, legacy 501 class.
Schema version assertions 3→4.
… in passing)

Master-side regression, reproduced on plain origin/master (c800ed1):
the #3067 merge (effectiveConfig embeds preserveRecentMessages 3)
landed after 8f93c33's regen, so the metadata section digest drifted
on all four wires; master's push CI skips the test matrix, so nobody
saw. State/blockContents/stats sections are byte-identical. Same
in-passing pattern as a937803.
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-14_storage-ext (928e484)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3518-pr.3070.101c8a8

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3070.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Starting review of #3070 (extension namespaces: session_ext + kv_store, schema v4).

Located the branch origin/2026-10-14_storage-ext, base c800ed1b (origin/master), 2 commits (bbbb4a487 feat + bf32bbb18 golden regen), 11 files / +487 -10. This is a persistence-format change (schema v3→v4) so it's human-merge territory by design.

Working through: original issue direction (#3069) → regression surface → code logic → test validity → diff cleanliness. Will post the structured verdict (eight questions) once done.

The v4 extension-namespaces feature (session_ext + kv_store) is tracked
in #3069; every provenance comment and the CHANGELOG line cited #3056,
which is an unrelated, already-closed codex inline auto-compact PR.
Correct the 10 references so grep-based archaeology lands on the right
issue. No logic change; comments/CHANGELOG only.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #3070 审核结论(extension namespaces / schema v4)

一句话判定:功能代码本身正确、可合;但分支在审核中途被改动——「顺手 golden 重生成」被 revert,当前 test lane 预期会红,需先确认 golden 状态再合。 人工合并必需(§7.4:持久化格式/版本变更 + feat)。

背景:给统一索引加两张通用表——session_ext(每会话任意状态,随会话删除)和 kv_store(命名空间跨会话状态,调用方管理生命周期),让 host/plugin/UI 功能在「full-default 翻转前的零包袱窗口」里有持久化落点,避免每个功能一次 schema 迁移。

结构化八问:

  1. 是什么问题:功能需求(非 bug)——为宿主/插件/UI 提供通用 KV 持久化,规避 per-feature 建表迁移。
  2. 什么原因导致:不适用(feature)。设计动机:逐功能建表会让迁移成本线性增长,通用命名空间把这件事推迟。附带项:master 侧 golden 漂移的根因是 fix: unblock master lanes — restore personaNamespace import + storage-render goldens after #3061 #3067 的 effectiveConfig 内嵌 preserveRecentMessages 晚于上次 golden 重生成落地。
  3. 影响范围与严重程度:纯增量。新库直接建 v4;旧 v3 库就地迁移(两条幂等 CREATE TABLE IF NOT EXISTS)。既有会话数据不动。新增 UnifiedStore/UnifiedShadow/SessionStore API。deleteSession 事务内多一条 DELETE FROM session_ext(无 ext 行的会话为 no-op)。无负面严重度。
  4. 怎么实现的:两张 WITHOUT ROWID 侧表放在 LegacySessionLike 视图之外(零 digest parity 扰动);guard(key≤256 且无控制字符、value≤4MiB JSON);upsert;前缀过滤对 LIKE 元字符(%/_/\)转义;session_ext 在删除事务内清扫;shadow/full 透传;legacy-only fail-fast(同 replica 路由的 501 文案)。迁移=既有事务里幂等建表。
  5. 是否有回归问题:逐条过 diff 触碰的既有行为——(a) deleteSession 事务:新增删除已确认在事务内、位于删 conversations 之后删 sessions 之前,其余删除未动,无 ext 行时 no-op;(b) STORAGE_SCHEMA_VERSION 3→4:所有引用收敛在 schema.ts,唯一断言 ==3 的用例(tests/storage-unified.test.ts:709)已正确改为 4,无其它破坏;(c) 唯一悬而未决项:PR 描述里的「顺手 golden 重生成」(bf32bbb18)在审核中途被 revert(daebe97be),重新暴露了 base c800ed1 上因 fix: unblock master lanes — restore personaNamespace import + storage-render goldens after #3061 #3067 造成的 storage-render 元数据 digest 漂移(state/blockContents/stats 字节一致)。revert 后本 PR 的 test lane 预期会红在 tests/golden-storage-render.test.ts,除非恢复重生成或先落配套 golden 修复。(我未本地复现——golden 存的是 digest 而非原始字段,且单测需全量 build;依据是你 PR 里的根因描述 + fix: unblock master lanes — restore personaNamespace import + storage-render goldens after #3061 #3067 已在 master。)(d) 功能代码本身无 fail-fast 变吞错、无阈值/时序/输出格式漂移。
  6. 是否破坏缓存:否。两表都在加载路径与 LegacySessionLike 视图之外,不动 digest parity、不动头部 system 前缀缓存;state/blockContents/stats golden 字节一致佐证。
  7. 是否改设计方向:没有转向,正是 issue 引用的既定扩展底座路线(Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671 §7 / Identity pipeline: key sessions by (agent, conversation id) in the unified store — decouple fold ledgers from protocol/host origin (#2938 / #2847 / #2671 defect-1) #2960 路线图)在 full-default 翻转前的一次通用化;内核状态刻意不迁入(kstate_json 不变)。
  8. 是否引入新配置:无新 config/env/CLI。guard 用内部常量(EXT_KEY_MAX=256、EXT_VALUE_MAX_BYTES=4MiB),非配置面;config-surface-guard lane 通过。

发现与处置:

次要观察(不阻断,未扩大范围):store.ts:173 注释写「printable ASCII-range」但 guard 实际允许非 ASCII(CJK 可过);value 上限按字符数而非字节计;LIKE 反斜杠转义用例未覆盖。均为可选打磨。

门禁状态(审核时点):static-gates / kernel-guard / pi-subagents-guard / config-surface-guard / version-guard / unused-exports / docs-* / build-artifact / 各 real-client e2e lane 全绿;test (ubuntu|windows × 22|24) 矩阵仍在跑(golden 结果待其落定)。

收工判定:代码批准;合并 hold——等 storage-render lane 绿(恢复重生成或先落配套 golden 修复)后再由人工合并。

@ranxianglei
ranxianglei merged commit b8e2144 into master Oct 14, 2026
14 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.390-master.b8e2144 (rolling master tag).

npm install -g billion-context@0.0.390-master.b8e2144  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extension namespaces: per-session arbitrary state + namespaced KV in the unified index (schema v4)

1 participant