Skip to content

fix: decouple e2e suites from repo AGENTS.md; pin codex; realistic fake usage - #830

Merged
ranxianglei merged 1 commit into
masterfrom
2026-09-16_e2e-hermetic-cwd
Sep 16, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-09-16_e2e-hermetic-cwd

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #815

Related: #829 (estimator gap found while analyzing this issue — tracked separately on purpose, NOT included here, see below)

What changed

  • tests/e2e/e2e-codex-fake.test.ts + tests/e2e/e2e-codex.test.ts: codex is now spawned with a cwd outside the repo tree ($TMPDIR/billion-context-e2e/cwd-*). codex discovers AGENTS.md by walking UP from its spawn cwd and concatenates every level into the system prompt (verified A/B locally on codex-cli 0.147.0: in-repo cwd → request 69,424 chars incl. repo doc; out-of-tree cwd → 44,380 chars, no doc marker). An empty stub AGENTS.md does NOT work — codex concatenates parent levels too. State dirs stay in <repo>/tmp/e2e-codex-fake-* because codex refuses TMPDIR homes and CI collects artifacts from there.
  • overflow window 12k → 10k (e2e-codex-fake.test.ts): the hermetic baseline measures ~14k reported tokens (codex base system prompt + bili injection), so the window must sit below it for preflight to engage early enough to keep the forwarded payload bounded — the same role the old 12k played against the AGENTS.md-inflated baseline.
  • .github/workflows/ci-e2e-fake.yml + ci-e2e.yml: pin @openai/codex@0.147.0 and print the version at runtime. Unpinned @latest let any codex release shift the baseline system-prompt size and break the deterministic overflow phase the same way a doc change did.
  • tests/e2e/fake-upstream.mjs: the stream path now reports estIn(raw) instead of a hardcoded inTok = 1 (a placeholder from the file's first commit; the non-stream path always used estIn). bili's gate consumes the reported usage, which was stuck at 1 for every streamed turn.
  • tests/e2e/README.md: documents the hermetic-cwd requirement.

Why this shape (and what is deliberately NOT here)

While reproducing #815 I found a second, production-relevant defect: estimateWireOverhead for the responses protocol never counts the developer/system items codex carries inside input[], so the gate undercounts every codex-shaped payload by its entire system prompt (~6–9k tokens measured). Fixing it shifts real users' preflight trigger timing and requires retuning ~13 unit tests whose windows were calibrated against the old incomplete accounting — that is #829, filed separately with repro data and a drafted fix. This PR stays scoped to CI reliability; the e2e suite passes without it (hermetic cwd + realistic fake usage are sufficient).

Verification

  • npm run typecheck ✓
  • npm test: 1513 tests, 0 fail ✓
  • npm run build ✓
  • ACP_TEST_E2E_FAKE=1 node --import tsx --test tests/e2e/e2e-codex-fake.test.ts — both tests pass, run twice (deterministic suite):
    • overflow: 3 summarization calls; forwarded mains 56,601 / 67,018 / 56,557 / 66,972 / 57,125 chars (peak/min = 1.19 ≤ 1.3); sentinels retained in final payload
    • control (60k window): no summarization calls
  • Real-upstream e2e (ACP_TEST_E2E) not run locally (needs E2E_UPSTREAM_URL); the change there is the identical, already-verified spawn-cwd pattern, and that suite is manual-dispatch only in CI.

Compression modes

Test-infrastructure-only change: no wire format, message rebuild, tool injection, or summary-carrier behavior touched. Neither plugin mode nor proxy mode is affected.

For #803

Once this merges, re-running #803's CI should go green without splitting: the AGENTS.md growth can no longer reach codex's prompt.

…ke usage

- e2e-fake + e2e-codex: spawn codex with a cwd outside the repo tree
  ($TMPDIR/billion-context-e2e-*). codex discovers AGENTS.md by walking UP
  from its spawn cwd and concatenates every level into the system prompt,
  so any repo-doc growth inflated every request and broke the
  deterministic overflow phase (#815). State dirs stay in-repo: codex
  refuses TMPDIR homes and CI collects artifacts from tmp/e2e-codex-*.
- overflow window 12k -> 10k: the hermetic baseline measures ~14k reported
  tokens, so the window must sit below it for preflight to engage early
  enough to keep the forwarded payload bounded (same role as the old 12k
  against the AGENTS.md-inflated baseline).
- ci-e2e*.yml: pin @openai/codex@0.147.0 and print the version at runtime;
  unpinned @latest let any codex release shift the baseline and break the
  suite the same way.
- fake-upstream.mjs: the stream path now reports estIn(raw) instead of a
  hardcoded 1 input token (matching the non-stream path); bili's gate
  consumed the reported usage, which was stuck at 1.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-16_e2e-hermetic-cwd (319c4c4)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-830

Each push to this PR publishes a new version under the pr-830 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr830.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Reviewing PR #830 now: pulling the branch, auditing the diff, running typecheck/tests/build locally, plus the fake-upstream e2e suite. Will report findings here shortly.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review complete — LGTM, ready to merge. I pulled the branch (319c4c4) and verified end to end, including against current master.

Diff audit — clean. 6 files, +41/−9, every line serves the stated purpose (hermetic spawn cwd ×2 suites, codex pin ×2 workflows, estIn fix in the fake upstream, README note). No unrelated churn, no lockfile drift, no version touch. YAML comment placement in both workflows parses fine. Duplicate scan: this is the only PR touching #815.

Local verification (codex-cli 0.147.0, matching the pin):

Design judgments:

Minor observations (non-blocking, no change requested):

  • teardown() doesn't remove the per-test $TMPDIR/billion-context-e2e/cwd-* dirs (or the in-repo work dirs, as before). Consistent with existing convention and OS-managed locations; fine to leave.
  • Module-load mkdirSync(CWD_ROOT) runs even when the suite skips — harmless idempotent dir creation, same pattern as the pre-existing WORK_ROOT.

Residual risk (low): the real-upstream suite (ci-e2e.yml, manual-dispatch) couldn't be exercised here (needs E2E_UPSTREAM_URL). Risk is low because its assertions are relative (usage grew / compression event / usage dropped / purity recall), not absolute-size based, and the hermetic baseline (~14k reported tokens) still crosses the 32k phase window around load turn 6 — inside Phase 3's "one opportunistic extra turn" budget, with nudge firing earlier in practice. The next manual-dispatch run will confirm; worth doing before or right after merge. For #803, agree: once merged, its CI re-run should go green without splitting.

中文摘要:修复了 e2e 套件与仓库 AGENTS.md 体积的隐性耦合(#815,spawn cwd 移出仓库树)并顺带修了 fake upstream 流式路径 usage 恒为 1 的占位缺陷;diff 干净、本地在 PR 分支和合并当前 master(含 acp-kernel 0.0.73)后全部验证通过(typecheck/build/单测/两次确定性 e2e),CI 全绿,没有需要我代改的小问题,可以合并。

@ranxianglei
ranxianglei merged commit 50f8f9f into master Sep 16, 2026
7 checks passed
@ranxianglei
ranxianglei deleted the 2026-09-16_e2e-hermetic-cwd branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[e2e] fake-upstream overflow 测试与仓库 AGENTS.md 体积隐性耦合:AGENTS.md 增长即破坏 CI(#803 触发)

1 participant