Skip to content

DSH plugin lane: unforwarded toolCallId yields synthetic plugin_* IDs, so obsolete compress exchanges are not retired #7239

Open

Description

@onestone1879

Summary

In the DSH plugin lane, compression itself can still reduce context, but obsolete compress call/result pairs may survive their intended lifecycle and raise the post-compression floor. The DSH adapter invokes forwardTool without the host toolCallId, so blocks get synthetic plugin_* IDs while replayed tool calls keep call_* IDs. hideConsumedCompressCalls then treats bare compress calls as orphans when IDs do not match, and a numeric-ref sweep cannot retire BLOCKED refs (refNum(BLOCKED) is null).

This is a DSH-lane recurrence of the mechanism fixed for Pi in #3587, not a request to reopen that issue, and not a request to drop protected history unconditionally. A fix must keep a complete summary carrier for every active block in both plugin and proxy modes (see #5866).

A separate parser issue, #7240, covers invalid newline-joined JSON tuples being accepted as a single range. The two defects were observed together in one incident and amplify each other; they should stay separate repair scopes.

Verified environment

  • Incident: billion-context 0.1.215, DSH 0.2.2-rc.3, plugin-native lane, OpenAI-compatible wire. DSH is deployed on Ubuntu (release not provided), using a self-hosted upstream serving qwen3.8-flash-next (reporter-confirmed).
  • The identity / BLOCKED branch was also reproduced offline at 0.1.216, commit 6a641f378537e06e7430ec90c9abf035ffa12377, with Node.js 24.
  • These tests are function-level synthetic probes, not a full DSH wire capture. Later releases, including current master, have not been rechecked here.

Related work (not duplicates)

Mechanism and code

Source links are pinned to the checked 0.1.216 commit; the implicated 0.1.215 logic was also inspected.

  1. The DSH adapter calls forwardTool(base, sid, tool.name, args, exec.signal, true) without the optional toolCallId.
  2. Plugin execution then creates a plugin_-prefixed synthetic ID when none is supplied.
  3. hideConsumedCompressCalls resolves bare compress by call ID; unmatched bare calls become orphans. A sweep requires a numeric ref, while refNum(BLOCKED) is null. This describes the tested branch, not a claim that a later range can never cover these messages.

Minimal synthetic reproduction

Function-level only: isolated checkout, Node.js 24, no production data, and no running DSH. Please attach reproduce.mjs, observed-0.1.215.json, and observed-0.1.216.json (the two result files are byte-identical). The issue-1 cases remain understandable from the table below if the files are missing.

git clone --depth 1 --branch v0.1.215 https://github.com/ranxianglei/billion-context.git bili-0215
node reproduce.mjs ./bili-0215

git clone https://github.com/ranxianglei/billion-context.git bili-0216
git -C bili-0216 checkout --detach 6a641f378537e06e7430ec90c9abf035ffa12377
node reproduce.mjs ./bili-0216

Results on both checked versions:

Fixture hidden kept notes
Inactive block, mismatching IDs, BLOCKED refs 0 2 headline case
Same fixture, matching IDs 2 0 diagnostic
Same fixture, mismatching IDs but numeric refs 2 0 diagnostic, not a proposed patch
Active matching ID with documented line-form content 0 2 (call unchanged) consistent with #3975; not claimed as new

A script failure after export or API changes on a newer revision is not evidence that the defect is gone. The ID/ref variants are diagnostic counterfactuals, not safe patches.

Sanitized incident evidence

Private session content is not attached. These aggregates come from a 0.1.215 + DSH 0.2.2-rc.3 OpenAI-compatible session:

  • 20 compress calls / 20 blocks; every block ID differed from the corresponding tool-call ID; 15 blocks stayed active.
  • 18 call/result pairs were outside later block coverage: 97,774 characters of call text plus 15,911 characters of results. Two other pairs were covered by a later range, so BLOCKED is not an absolute barrier to later coverage.
  • Five inactive blocks had uncovered original calls totaling 20,032 call-text characters.
  • Offline replay of those 18 uncovered calls against actual block metadata retained all 36 messages.
  • A carrier-strip helper comparison retained 15 summary anchors (61,358 summary characters) with the actual IDs, versus two anchors (5,822 characters) in a diagnostic ID-matched run — a 55,536-character duplicate-carrier opportunity in that replay, not a measured historical wire or token saving.
  • Actual upstream input usage once fell from 120,630 to 67,328 tokens, so ordinary compression was not completely ineffective. Later post-fold baselines rose; the last available raw input usage was 142,244 tokens. A session-accounting display of about 385,104 tokens is not evidence of a 385k outbound request.

Why the two issues together are serious

On their own, leftover identity mismatches keep obsolete compress machinery in context. In the same incident, invalid multiline JSON-tuple content was silently accepted as one range (#7240), so long summary text moved into topic and batch execution was easy to over-count. Combined, oversized topics and unretired call/result text can raise the post-compression floor even when a fold still reduces usage. This is not a claim of irreversible data loss, and not causal proof that the later 142,244 baseline was produced only by these two defects.

Expected behavior

Suggested labels: bug, p1, dsh, scope/kernel (the omitted toolCallId is in the DSH adapter; maintainers may also want scope/wiring). Theme: C (compression engine) and F (dsh lane).

Appendix: synthetic reproduction files

GitHub's issue API cannot attach files, so the three approved synthetic files are included below verbatim. observed-0.1.215.json and observed-0.1.216.json are byte-identical.

SHA-256:

  • reproduce.mjs: df04e74c77c1c25382a1aef6a85771d806dc2c3511ab508fcaeed8112310979f
  • observed-0.1.215.json / observed-0.1.216.json: 0f6ca47adb3ae63a0d2159004bc6e436191fb34bc3d006fd78f448c8c3c39c06

reproduce.mjs

// Synthetic fixtures only. Reads an isolated upstream checkout; no server or database access.
import { registerHooks } from 'node:module';
import { existsSync } from 'node:fs';
import { resolve, sep } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import assert from 'node:assert/strict';
const input = process.argv[2];
if (!input) throw new Error('Usage: node reproduce.mjs /path/to/isolated/billion-context-checkout');
const root = resolve(input);
registerHooks({ resolve(specifier, context, nextResolve) {
  if (specifier.startsWith('.') && specifier.endsWith('.js') && context.parentURL?.startsWith('file:')) {
    const parent = fileURLToPath(context.parentURL);
    if (parent.startsWith(root + sep)) {
      const candidate = new URL(specifier.slice(0, -3) + '.ts', context.parentURL);
      if (existsSync(fileURLToPath(candidate))) return nextResolve(candidate.href, context);
    }
  }
  return nextResolve(specifier, context);
}});
const load = relative => import(pathToFileURL(resolve(root, relative)).href);
const { parseCompressArgs } = await load('kernel/src/parse-compress-input.ts');
const { hideConsumedCompressCalls } = await load('kernel/src/hide-consumed.ts');
const malformed = { content: '["m00001–m00002","TITLE ONE","' + 'A'.repeat(24000) + '"]\n["m00003–m00004","TITLE TWO","BODY TWO"]' };
const summarize = p => ({ ok: p.diagnostics.ok, ranges: p.ranges.map(r => ({ startRef: r.startRef, endRef: r.endRef, topicChars: r.topic?.length ?? 0, summaryChars: r.summary.length })) });
const correct = { content: 'm00001–m00002 First\nSummary one.\n\nm00003–m00004 Second\nSummary two.' };
const good = parseCompressArgs(correct);
assert.equal(good.ranges.length, 2, 'Control: documented line format should parse two ranges');
const call = { id: 'synthetic-message-call', role: 'assistant', contentType: 'tool-call', toolName: 'compress', toolCallId: 'call_synthetic', text: JSON.stringify({ content: 'm00001–m00002 Short topic\n' + 'S'.repeat(1000) }) };
const result = { id: 'synthetic-message-result', role: 'tool', contentType: 'tool-result', toolName: 'compress', toolCallId: 'call_synthetic', text: 'Synthetic successful receipt' };
const block = { blockId: 'b1', startRef: 'm00001', endRef: 'm00002', compressCallId: 'plugin_synthetic', active: false, topic: 'Short topic', summary: 'S'.repeat(1000) };
function run({ matchedId = false, numericRefs = false, active = false } = {}) {
  const state = { blocks: [{ ...block, active, compressCallId: matchedId ? call.toolCallId : block.compressCallId }], messageRefs: { byRaw: { [call.id]: numericRefs ? 'm00003' : 'BLOCKED', [result.id]: numericRefs ? 'm00004' : 'BLOCKED' }, byRef: {} }, pendingOrphanSweepMinRef: 1 };
  const out = hideConsumedCompressCalls(state, [call, result]);
  return { hidden: out.hidden, kept: out.messages.length, callUnchanged: out.messages.some(m => m.id === call.id && m.text === call.text) };
}
console.log(JSON.stringify({ fixture: 'Synthetic, function-level only; no historical wire capture and no carrier-strip E2E assertion.', issue1: { inactiveIdMismatchBlocked: run(), inactiveIdMatchedBlocked: run({ matchedId: true }), inactiveIdMismatchNumeric: run({ numericRefs: true }), activeIdMatchedLineForm: run({ matchedId: true, active: true }) }, issue2: { malformed: summarize(parseCompressArgs(malformed)), correctLineForm: summarize(good), nestedTupleDiagnostics: parseCompressArgs({ content: JSON.stringify([['m00001–m00002', 'Title', 'Summary']]) }).diagnostics } }, null, 2));

observed-0.1.215.json

{
  "fixture": "Synthetic, function-level only; no historical wire capture and no carrier-strip E2E assertion.",
  "issue1": {
    "inactiveIdMismatchBlocked": {
      "hidden": 0,
      "kept": 2,
      "callUnchanged": true
    },
    "inactiveIdMatchedBlocked": {
      "hidden": 2,
      "kept": 0,
      "callUnchanged": false
    },
    "inactiveIdMismatchNumeric": {
      "hidden": 2,
      "kept": 0,
      "callUnchanged": false
    },
    "activeIdMatchedLineForm": {
      "hidden": 0,
      "kept": 2,
      "callUnchanged": true
    }
  },
  "issue2": {
    "malformed": {
      "ok": true,
      "ranges": [
        {
          "startRef": "m00001",
          "endRef": "m00002",
          "topicChars": 24017,
          "summaryChars": 38
        }
      ]
    },
    "correctLineForm": {
      "ok": true,
      "ranges": [
        {
          "startRef": "m00001",
          "endRef": "m00002",
          "topicChars": 5,
          "summaryChars": 12
        },
        {
          "startRef": "m00003",
          "endRef": "m00004",
          "topicChars": 6,
          "summaryChars": 12
        }
      ]
    },
    "nestedTupleDiagnostics": {
      "ok": false,
      "kind": "no-valid-ranges",
      "invalidItems": 1,
      "keys": [
        "content"
      ],
      "invalidReasons": [
        "entry 0: not an object"
      ]
    }
  }
}

observed-0.1.216.json

This file is byte-identical to observed-0.1.215.json. The same bytes are repeated here so the filename is present.

{
  "fixture": "Synthetic, function-level only; no historical wire capture and no carrier-strip E2E assertion.",
  "issue1": {
    "inactiveIdMismatchBlocked": {
      "hidden": 0,
      "kept": 2,
      "callUnchanged": true
    },
    "inactiveIdMatchedBlocked": {
      "hidden": 2,
      "kept": 0,
      "callUnchanged": false
    },
    "inactiveIdMismatchNumeric": {
      "hidden": 2,
      "kept": 0,
      "callUnchanged": false
    },
    "activeIdMatchedLineForm": {
      "hidden": 0,
      "kept": 2,
      "callUnchanged": true
    }
  },
  "issue2": {
    "malformed": {
      "ok": true,
      "ranges": [
        {
          "startRef": "m00001",
          "endRef": "m00002",
          "topicChars": 24017,
          "summaryChars": 38
        }
      ]
    },
    "correctLineForm": {
      "ok": true,
      "ranges": [
        {
          "startRef": "m00001",
          "endRef": "m00002",
          "topicChars": 5,
          "summaryChars": 12
        },
        {
          "startRef": "m00003",
          "endRef": "m00004",
          "topicChars": 6,
          "summaryChars": 12
        }
      ]
    },
    "nestedTupleDiagnostics": {
      "ok": false,
      "kind": "no-valid-ranges",
      "invalidItems": 1,
      "keys": [
        "content"
      ],
      "invalidReasons": [
        "entry 0: not an object"
      ]
    }
  }
}

Activity

  1. ranxianglei commented on Nov 19, 2026

    @ranxianglei
    Owner

    (已收)本单已归入根因聚合 issue(token 见聚合单 body),机器人已停止在本单上的工作,修复走聚合 issue 统一出 PR;本单保留不关,修复验证后统一关闭。(聚合单:[Root Cause] dsh-lane (10 items),GitHub #6446)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions