Skip to content

[PR] feat: sticky 10% lean prompt-pack canary for unset compress.promptPack (#1408) - #1433

Closed
ranxianglei wants to merge 6 commits into
masterfrom
2026-09-26_prompt-pack-canary
Closed

ranxianglei wants to merge 6 commits into
masterfrom
2026-09-26_prompt-pack-canary

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #1408

Behavior change (intended — owner decision in #1408)

Old: unset compress.promptPack → identity surface (default pack) for every session, always.
New: unset compress.promptPack → each session is assigned a builtin pack once and keeps it for its whole life: ~10% of sessions (deterministic session-id hash) get the builtin lean pack, the rest default. Rollout knob: BILI_PROMPT_PACK_CANARY_PCT (integer 0–100, default 10; non-numeric/out-of-range → 10; 0 = off, 100 = full lean).

Why: owner decision — "default becomes lean — via a 10% canary first", with an observation window afterwards before deciding 100 vs rollback to 0.

Assignment semantics: persisted assignment, not per-request hash

The constraint set includes both "once lean always lean" AND "once default stays default". A pure per-request hash(sessionId) % 100 < pct violates the second half whenever pct moves in either direction (raising it flips existing default-sessions to lean mid-experiment; rolling back to 0 yanks lean out of already-assigned sessions mid-observation). So the assignment is computed ONCE — on the first request where promptPack is unset at every level — against the CURRENT pct, stamped onto session.meta.packCanary, persisted with the session, and never re-evaluated:

  • raising pct: only not-yet-assigned sessions become eligible (at their own hash); no existing session flips;
  • lowering pct / rollback to 0: stops NEW lean assignments; already-lean sessions stay lean for their whole life (documented as intended — rollback takes effect on new sessions);
  • restarts / proxy-instance moves: stamp loads from disk — no re-roll, no drift;
  • any explicit promptPack — including an explicit "default" — always wins and is never canaried (canary engages only when the merged value is undefined);
  • eligibility itself stays a deterministic id-hash (sha256(sessionId) bucket % 100 < pct): stateless at decision time, reproducible across instances.

Observability

  • one info log at first assignment: prompt-pack canary: lean (pct=10, session=<id>) (also logged for default assignments, so the split is auditable);
  • the canaried value flows through the existing activePack audit stamp, so the /acp panel active-pack line shows pack=lean for canaried sessions exactly like an explicit setting (verified by test).

Drive-by fix (own labeled commit)

fix: restore session.meta.activePack on session reload — buildRecord wrote meta.activePack to disk (meta: {...session.meta}) but buildSession never restored it, so after a restart the /acp panel showed pack=default for every reloaded session until its next request re-stamped it — contradicting the server.ts comment "Persisted with the session". Restored on load. Minor behavior change of its own: post-restart panel now shows the last known pack instead of default until the next request. Found while adding meta.packCanary persistence in the same code path; pinned by extending the disk-round-trip test.

Docs (own commit)

CONFIGURATION.md + zh-CN mirror: promptPack Default line, new "Sticky lean canary" paragraph, BILI_PROMPT_PACK_CANARY_PCT env-var table row. Also corrects the builtin-lean description: on the pi host the pack additionally ships condensed system-prompt sections via its pi adapter (compact acp-tags/summary-context guidance + shorter how-to-compress) — previously documented as "keeping the compression rules default" everywhere.

Tests

New tests/prompt-pack-canary.test.ts (11 tests) pins everything requested in the issue:

  • unset + hash-eligible → lean surface reaches the outbound OpenAI wire (tool description asserted byte-exact)
  • unset + hash-ineligible → kernel-default surface
  • explicit route-level promptPack: "default" → never canaried even at pct=100
  • pct=0 disables (unset → default); pct=100 full lean; invalid pct ("banana") → behaves as default 10
  • knob parsing table (unset / whitespace / negative / >100 / non-integer → 10)
  • eligibility determinism + monotonicity + ~10% distribution over 2000 ids
  • stickiness: knob moves mid-process never re-roll assigned sessions
  • disk round-trip: stamp survives a restart through a real SessionStore (assignment + audit stamp)

Pre-flight on this head: typecheck ✓, full suite ✓ (2458 pass; the 3 claude-native.test.ts failures are pre-existing on clean master in this sandbox — process-spawn env issues, verified by stashing the branch), build ✓. Note: this touches the src/server.ts request pipeline; the codex e2e suite (ACP_TEST_E2E=1) needs a local upstream that this sandbox cannot reach — worth a manual-dispatch CI e2e run pre-merge.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Reviewing PR #1433 (sticky 10% lean prompt-pack canary, Closes #1408). Initial state check done:

  • Branch 2026-09-26_prompt-pack-canary fetched (commits a06b1133 feat + b7d458c6 drive-by activePack-reload fix + 8493af19 docs; 7 files, +433/−5).
  • ⚠️ Stale base: branch is off master @ 088bb348 (v0.1.147 era); master has since moved to 825f1529 (v0.1.158, ~11 releases incl. kernel pins 0.0.86→0.0.95, CCR/acp_rule default changes, heavy src/server.ts churn). GitHub reports mergeable=false (dirty).
  • No CI runs recorded on head SHA yet (PR opened minutes ago).

Per §7.2 I'm rebasing onto current master, resolving any conflicts (hot files: src/server.ts, src/persist.ts, src/session.ts), re-running typecheck + full suite + build on the rebased head, then reporting the verdict here. Full review (direction vs #1408, regression audit, test validity) follows.

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-26_prompt-pack-canary (2985fd7)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1745-pr.1433.72515c7

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1433.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review complete — branch rebased onto current master (825f1529, v0.1.158) and force-pushed to 2026-09-26_prompt-pack-canary (new head e7961e08). The PR was stuck at mergeable: false/dirty on a stale base (merge-base was v0.1.147; master had moved ~70 commits ahead incl. acp-kernel pin 0.0.86→0.0.95). It is now mergeable: true and all 8 CI workflows (CI ×2 platforms, real/fake codex e2e, native pi/opencode/dsh e2e, local-registry, build artifact) are running on the new head.

Conflicts resolved (union of intent, verified per-file):

  1. src/server.ts — master had inserted the [#1207 owner decision] CCR opt-in comment block exactly where this PR adds mergedCompressCfg/promptPackUnset; both kept, canary block placement unchanged (right after getSession, before the feat(compress): promptPack — kernel pack resolution wired through all three wire paths #730 audit stamp).
    2+3. CONFIGURATION.md / CONFIGURATION.zh-CN.md env table — master added 4 rows since the base (BILI_CHAIN_CONTENT rewritten for bug(chain): ACP tag literals in normal history can permanently force fresh proxy sessions into passthrough #1357 advisory-only semantics, plus BILI_CHAIN_MAX_FUTURE_SKEW_MS, BILI_CHAIN_RECENT_WINDOW_MS, BILI_CONFLICT_SCAN). The PR's copy of the old BILI_CHAIN_CONTENT row was stale and dropped; all master rows kept verbatim + the new BILI_PROMPT_PACK_CANARY_PCT row appended. zh/en stay in sync.
  2. Rebase-introduced breakage found & fixed locally: the PR's hunk appended type CompressSettings to the second ./config.js import line, but master already imports it on line 9 (import type { CompressSettings, ProxyOptions }) → TS2300 duplicate identifier. Removed the duplicate and autosquashed it into the feat commit — commit structure stays clean (feat / fix / docs).

Regression audit (existing behaviors on the touched paths):

Behavior change (intended, per owner decision in #1408): unset compress.promptPack used to mean the default pack for every session; now each such session gets a builtin pack assigned once and persisted (session.meta.packCanary): deterministic sha256(sessionId) bucket % 100 < BILI_PROMPT_PACK_CANARY_PCT (default 10), so ~10% get lean. Moving the knob never re-rolls assigned sessions (0 stops new lean assignments, 100 admits all not-yet-assigned); any explicit setting including "default" always wins. Drive-by fix disclosed in the PR body: post-restart /acp panel now shows the last-known pack instead of resetting to default until the next request.

Pre-flight on the rebased head: typecheck ✓ (clean after the import fix), full test suite ✓ (2850 pass / 0 fail / 12 skipped of 2862 — the 3 claude-native spawn-env failures seen earlier in this sandbox did not reproduce on this tree), build ✓ (dist-import-annotations: OK). The real-codex e2e phase is part of this push's CI run — I'll confirm its result shortly.

Not merging — that stays with you. Once CI is green, this is ready to merge.

已将分支 rebase 到最新 master(解决 server.ts 与两份配置文档的冲突,并修掉 rebase 引入的重复类型导入),本地 typecheck、测试(2850 通过 0 失败)、构建全部通过,PR 现已可合并、CI 正在运行,CI 绿后即可人工合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI update: all 9 workflows are now green on head e7961e08 — CI (ubuntu+windows matrix), CI Windows codex overlay (#686), E2E (real codex) ✅, e2e-fake-codex ✅, e2e-native-pi / e2e-native-opencode / e2e-native-dsh / e2e-local-registry, and PR Build Artifact. The real-codex e2e phase that I could not reach from this sandbox ran in CI and passed, so the last open verification item from the PR body is closed.

Nothing further outstanding on my side: code reviewed against current master, regression audit clean, behavior change disclosed above, pre-flight + full CI green, PR mergeable. Ready for you to merge whenever.

CI 已全部通过(含真实 codex e2e),无遗留验证项,PR 已就绪,等你人工合并。

ework-agent and others added 4 commits September 27, 2026 09:31
#1408)

When compress.promptPack is unset at every level, each session is assigned a builtin pack once by deterministic session-id hash against BILI_PROMPT_PACK_CANARY_PCT (default 10, validated 0-100, non-numeric -> 10) and keeps it for its whole life: the choice is stamped onto session.meta.packCanary (persisted with the session) and never re-evaluated, so knob moves never re-roll an existing session - raising pct admits not-yet-assigned sessions, 0 stops new lean assignments without flipping assigned ones. Any explicit promptPack (including an explicit "default") always wins and is never canaried. One info log at first assignment; the canaried value flows into the existing activePack audit stamp so the /acp panel shows it.
…amp)

buildRecord persisted meta.activePack to disk (meta: {...session.meta}) but buildSession never restored it, so after a restart the /acp panel showed pack=default for every reloaded session until its next request re-stamped it - contradicting the server.ts comment that the stamp is 'persisted with the session'. Restore it on load so post-restart forensics and the canary's observability (constraint 4 of #1408) hold across restarts. Found while adding meta.packCanary persistence for the canary.
CONFIGURATION.md (+ zh-CN mirror): promptPack Default line now points at the canary, a 'Sticky lean canary' paragraph documents the assign-once semantics (explicit settings never canaried, knob moves never re-roll, 0 stops new assignments, 100 admits all unassigned), and BILI_PROMPT_PACK_CANARY_PCT joins the env-var table. Also corrects the builtin-lean description: on the pi host the pack additionally ships condensed system-prompt sections via its pi adapter (compact acp-tags/summary-context guidance + shorter how-to-compress) - previously documented as 'keeping the compression rules default' everywhere.
… name (#1408 review F1/F2/F3)

- F1/F2: assignment previously keyed on 'first request whose pack is unset',
  so ~10% of EXISTING long-running sessions (no stamp, created before the
  rollout) and sessions whose explicit promptPack was later REMOVED flipped
  default->lean mid-conversation on their next request — contradicting the
  no-flip claim. New assignments are now birth-gated: only a session whose
  stats.requests is still 0 (the very first request; all ++ sites live later
  in the pipeline) may be assigned. The stamp READ stays on every request, so
  assigned sessions keep their pack regardless. Unstamped already-ran
  sessions resolve 'default' without writing a stamp.
- resolveStickyPackAssignment gains allowNewAssignment (default true).
- F3: document that 'lean' refers to the builtin pack — a user-authored
  packs/lean.json shadows it through the normal resolution chain; treat the
  name as reserved (CONFIGURATION.md + zh-CN).
- New e2e: explicit-pack turn 1, setting removed, pct=100 — session stays
  default, no stamp ever written.
@ranxianglei
ranxianglei force-pushed the 2026-09-26_prompt-pack-canary branch from e7961e0 to 0f0e38d Compare September 27, 2026 01:36
@ranxianglei

Copy link
Copy Markdown
Owner Author

Deep-review complete — Approve with F1/F2/F3 fixed on-branch (rebased onto current master: the PR branch contained master-landed pricing commits, so the canary commits were cherry-picked clean onto master + one review commit).

Verified

  1. Bucketing: sha256(sessionId) bucket %100, deterministic across restarts/instances; monotonicity + distribution matrix verified (11 existing tests).
  2. Default-off users byte-identical: compress tool schema shape-compare identical; prompts/thresholds/nudge cadence untouched by the pack surface; adapters field consumed nowhere in bili src.
  3. Canary = descriptions only: schema params identical (626→91 char description swap); lean surface never reaches plugin-mode models (tool injection suppressed).
  4. Observability: meta.packCanary + meta.activePack persisted, restored on reload (bdc1e27), first-assignment info log.
  5. Explicit always wins incl. explicit "default"; tested at route level pct=100.

Fixed on-branch

F1/F2 (low, design — deploy/config-removal flip): assignment keyed on first request whose pack is unset, so ~10% of EXISTING sessions (created pre-rollout, no stamp) — and sessions whose explicit promptPack was later removed — flipped default→lean mid-conversation on their next request, contradicting the no-flip claim. Fix: new assignments are birth-gated (stats.requests === 0; all increment sites live later in the pipeline, so this identifies the session's very first request). The stamp READ stays on every request — assigned sessions keep their pack forever; unstamped already-ran sessions resolve default without ever writing a stamp. resolveStickyPackAssignment gains allowNewAssignment (default true — unit callers unchanged).

F3 (low): documented that lean refers to the BUILTIN pack — a user-authored packs/lean.json shadows it through the normal resolution chain; the name is now documented as reserved (en+zh).

New e2e: explicit-pack turn 1 → setting removed → pct=100 → session stays default, meta.packCanary never written. All 12/12 green on-branch (typecheck clean; golden failures under the symlinked 0.0.93 node_modules are environment-only — CI installs the pinned 0.0.95).

Info (no action)

  • Raising pct admits only never-stamped sessions — widening for existing sessions requires deleting session files (now strengthened by the birth gate: only sessions BORN during the rollout can ever be assigned). Documented.
  • Plugin-mode sessions stamp activePack="lean" although the surface never reaches the model — forensics caveat, acceptable.
  • One info log per unset-pack session (incl. the 90% default).

Ready to merge once CI is green. Fixes #1408.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Verified the rebased branch end-to-end — all green, ready for you to merge.

State

Independent check of the review commit (0f0e38de)

  • F1/F2 birth-gate invariant verified by grepping every site: the decision point is src/server.ts:1764 (session.stats.requests === 0 passed as allowNewAssignment); ALL five ++session.stats.requests sites (src/server.ts:2743, 2980, 3240, 3429, 3811) plus the runPrepare compensating decrement (:4467) sit strictly later in the pipeline, and getSession initializes requests: 0 (src/session.ts:377) without incrementing. So === 0 at the decision point ⟺ the session's very first request — the claim holds. It also matches the pre-existing first-request-detection pattern used at server.ts:1803/1832/1879 (fork lineage, codex identity, conflict scan), so the gate rides established semantics rather than inventing a new one.
  • Stamp-read-first ordering confirmed in resolveStickyPackAssignment: stamped value returned before the gate is consulted → assigned sessions keep their pack forever regardless of knob or gate; unstamped already-ran sessions return "default" with no stamp written (the !allowNewAssignment early-return precedes the hash/stamp/log block). Unit callers unaffected (allowNewAssignment defaults true).
  • One benign concurrency edge: two genuinely parallel first requests could both see requests === 0 and both assign — but assignment is a deterministic id-hash, so both stamps are identical (idempotent write); worst case is a duplicated info log line. Not worth gating on.
  • New e2e (t12) does have teeth: explicit-pack turn 1 → setting removed → pct=100 → asserts the outbound wire carries the byte-exact kernel-default compress description AND that meta.packCanary is undefined in the persisted envelope via a real SessionStore round-trip through a fresh store instance. That pins both halves of the no-flip guarantee (behavior + persistence). Test count now 12 (was 11) ✓.
  • F3 docs: en+zh mirror in sync — birth-gate wording ("仅在它的第一个请求时…升级不会把任何活会话 default→lean 中途翻转") and the reserved-name caveat (packs/lean.json shadows the builtin through the normal chain; treat lean as reserved) both present in CONFIGURATION.md and CONFIGURATION.zh-CN.md.
  • Nit (non-blocking): the review commit carries one whitespace-only alignment fix (the resolvedImageCompressionCfg line re-indented 8→12 spaces to match its siblings). Harmless; noting it for diff-honesty only.

Regression surface re-checked against the advanced base: the canary block still sits between getSession and the #730 audit stamp (so the panel shows the canaried pack), #1207 CCR opt-in and #1357 advisory-chain doc lines from the earlier conflict resolution are intact, and nothing in #1285/#1419 touches the touched paths semantically (CI's full matrix + e2e confirms).

Not merging — that stays with you.

中文摘要:已独立核对你在分支上的 F1/F2/F3 修复(出生门控的增量点审计、stamp 优先读、新 e2e 断言有牙、中英文档同步),当前 head 0f0e38de 全部 9 条 CI(含真实 codex e2e)绿,可以合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

陈旧票清理(第三批动作①):目标 issue #1408 已关。为避免重复劳动关闭本 PR;如判断有误请直接 reopen 并留言。

@ranxianglei ranxianglei changed the title feat: sticky 10% lean prompt-pack canary for unset compress.promptPack (#1408) [PR] feat: sticky 10% lean prompt-pack canary for unset compress.promptPack (#1408) Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

默认配置为lean模式?

1 participant