Repository navigation
fix: decouple e2e suites from repo AGENTS.md; pin codex; realistic fake usage - #830
Conversation
…ke usage - e2e-fake + e2e-codex: spawn codex with a cwd outside the repo tree ($TMPDIR/billion-context-e2e-*). codex discovers AGENTS.md by walking UP from its spawn cwd and concatenates every level into the system prompt, so any repo-doc growth inflated every request and broke the deterministic overflow phase (#815). State dirs stay in-repo: codex refuses TMPDIR homes and CI collects artifacts from tmp/e2e-codex-*. - overflow window 12k -> 10k: the hermetic baseline measures ~14k reported tokens, so the window must sit below it for preflight to engage early enough to keep the forwarded payload bounded (same role as the old 12k against the AGENTS.md-inflated baseline). - ci-e2e*.yml: pin @openai/codex@0.147.0 and print the version at runtime; unpinned @latest let any codex release shift the baseline and break the suite the same way. - fake-upstream.mjs: the stream path now reports estIn(raw) instead of a hardcoded 1 input token (matching the non-stream path); bili's gate consumed the reported usage, which was stuck at 1.
📦 Built Package ArtifactBranch: Option A — Install from npm PR tag (recommended)npm install -g billion-context@pr-830Each push to this PR publishes a new version under the Option B — Download artifact
tar xzf billion-context-pr830.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 Review complete — LGTM, ready to merge. I pulled the branch ( Diff audit — clean. 6 files, +41/−9, every line serves the stated purpose (hermetic spawn cwd ×2 suites, codex pin ×2 workflows, Local verification (codex-cli 0.147.0, matching the pin):
Design judgments:
Minor observations (non-blocking, no change requested):
Residual risk (low): the real-upstream suite ( 中文摘要:修复了 e2e 套件与仓库 AGENTS.md 体积的隐性耦合(#815,spawn cwd 移出仓库树)并顺带修了 fake upstream 流式路径 usage 恒为 1 的占位缺陷;diff 干净、本地在 PR 分支和合并当前 master(含 acp-kernel 0.0.73)后全部验证通过(typecheck/build/单测/两次确定性 e2e),CI 全绿,没有需要我代改的小问题,可以合并。 |
Fixes #815
Related: #829 (estimator gap found while analyzing this issue — tracked separately on purpose, NOT included here, see below)
What changed
tests/e2e/e2e-codex-fake.test.ts+tests/e2e/e2e-codex.test.ts: codex is now spawned with a cwd outside the repo tree ($TMPDIR/billion-context-e2e/cwd-*). codex discoversAGENTS.mdby walking UP from its spawn cwd and concatenates every level into the system prompt (verified A/B locally on codex-cli 0.147.0: in-repo cwd → request 69,424 chars incl. repo doc; out-of-tree cwd → 44,380 chars, no doc marker). An empty stubAGENTS.mddoes NOT work — codex concatenates parent levels too. State dirs stay in<repo>/tmp/e2e-codex-fake-*because codex refuses TMPDIR homes and CI collects artifacts from there.e2e-codex-fake.test.ts): the hermetic baseline measures ~14k reported tokens (codex base system prompt + bili injection), so the window must sit below it for preflight to engage early enough to keep the forwarded payload bounded — the same role the old 12k played against the AGENTS.md-inflated baseline..github/workflows/ci-e2e-fake.yml+ci-e2e.yml: pin@openai/codex@0.147.0and print the version at runtime. Unpinned@latestlet any codex release shift the baseline system-prompt size and break the deterministic overflow phase the same way a doc change did.tests/e2e/fake-upstream.mjs: the stream path now reportsestIn(raw)instead of a hardcodedinTok = 1(a placeholder from the file's first commit; the non-stream path always usedestIn). bili's gate consumes the reported usage, which was stuck at 1 for every streamed turn.tests/e2e/README.md: documents the hermetic-cwd requirement.Why this shape (and what is deliberately NOT here)
While reproducing #815 I found a second, production-relevant defect:
estimateWireOverheadfor the responses protocol never counts the developer/system items codex carries insideinput[], so the gate undercounts every codex-shaped payload by its entire system prompt (~6–9k tokens measured). Fixing it shifts real users' preflight trigger timing and requires retuning ~13 unit tests whose windows were calibrated against the old incomplete accounting — that is #829, filed separately with repro data and a drafted fix. This PR stays scoped to CI reliability; the e2e suite passes without it (hermetic cwd + realistic fake usage are sufficient).Verification
npm run typecheck✓npm test: 1513 tests, 0 fail ✓npm run build✓ACP_TEST_E2E_FAKE=1 node --import tsx --test tests/e2e/e2e-codex-fake.test.ts— both tests pass, run twice (deterministic suite):ACP_TEST_E2E) not run locally (needsE2E_UPSTREAM_URL); the change there is the identical, already-verified spawn-cwd pattern, and that suite is manual-dispatch only in CI.Compression modes
Test-infrastructure-only change: no wire format, message rebuild, tool injection, or summary-carrier behavior touched. Neither plugin mode nor proxy mode is affected.
For #803
Once this merges, re-running #803's CI should go green without splitting: the AGENTS.md growth can no longer reach codex's prompt.