Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .github/workflows/ci-e2e.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: E2E (real codex)

on:
workflow_dispatch:
inputs:
forge:
description: "run forge phase (needs BILI_CODEX_COMPACT support in master)"
type: boolean
default: false

jobs:
e2e:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- name: Install codex CLI
run: npm install -g @openai/codex@latest
- run: npm ci
- run: npm run build
- name: Preflight
run: E2E_CHECK=1 node --import tsx --test tests/e2e/e2e-codex.test.ts
env:
E2E_UPSTREAM_URL: ${{ secrets.E2E_UPSTREAM_URL }}
E2E_UPSTREAM_KEY: ${{ secrets.E2E_UPSTREAM_KEY }}
- name: E2E suite
run: ACP_TEST_E2E=1 node --import tsx --test tests/e2e/e2e-codex.test.ts
env:
E2E_UPSTREAM_URL: ${{ secrets.E2E_UPSTREAM_URL }}
E2E_UPSTREAM_KEY: ${{ secrets.E2E_UPSTREAM_KEY }}
E2E_FORGE: ${{ inputs.forge && '1' || '' }}
- name: Collect artifacts
if: always()
run: |
mkdir -p artifacts && cp -r tmp/e2e-codex-* artifacts/ 2>/dev/null || true
- uses: actions/upload-artifact@v4
if: always()
with:
name: e2e-logs
path: artifacts/
retention-days: 7
35 changes: 35 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,41 @@ because `package.json` has proper `bin` + `files` fields. Either approach is
fine; the registry install is preferred for testing the real published
artifact.

### E2E Regression (real client through bili)

`tests/e2e/e2e-codex.test.ts` drives the **real `codex` CLI** through a built
proxy against a real Responses-compatible upstream and asserts the full
context lifecycle end-to-end: warmup → load growth → ACP compress → purity →
native-compact interception (last phase gated by `E2E_FORGE=1`). Full phase
details, env vars, and mechanics: `tests/e2e/README.md`.

```bash
# zero-token preflight (codex binary + dist + upstream reachable)
E2E_CHECK=1 node --import tsx --test tests/e2e/e2e-codex.test.ts

# full run (defaults to local sglang at http://127.0.0.1:8199/v1, zero cost)
npm run build
ACP_TEST_E2E=1 node --import tsx --test tests/e2e/e2e-codex.test.ts

# + native-compact interception phase
ACP_TEST_E2E=1 E2E_FORGE=1 node --import tsx --test tests/e2e/e2e-codex.test.ts
```

Rules:

- The suite **skips by default** so `npm test` stays free; never remove the
`ACP_TEST_E2E` gate.
- Run it (at least the 4-phase core) before merging changes to the request
pipeline — `server.ts`, `src/loop/`, adapters, preflight/compact paths.
It is the only coverage that exercises real client behavior (codex UA,
wire quirks, retry loops).
- Any Responses-compatible upstream works via `E2E_UPSTREAM_URL` /
`E2E_UPSTREAM_KEY`; the provider is configured as `name = "OpenAI"` so codex
stays on the remote compaction (V2) path — do not "fix" this.
- CI (`.github/workflows/ci-e2e.yml`) is manual-dispatch only and needs repo
secrets `E2E_UPSTREAM_URL` / `E2E_UPSTREAM_KEY`; a hosted runner cannot
reach `127.0.0.1` upstreams.

### Code Quality

- **No `as any`**, **No `@ts-ignore`**
Expand Down
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
"registry:snapshot": "node scripts/update-registry-snapshot.mjs",
"typecheck": "tsc --noEmit --project tsconfig.build.json",
"test": "node --import tsx --test tests/*.test.ts",
"start": "node dist/index.js"
"start": "node dist/index.js",
"test:e2e": "node --import tsx --test tests/e2e/e2e-codex.test.ts"
},
"keywords": [
"context",
Expand Down
71 changes: 71 additions & 0 deletions tests/e2e/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# E2E: real codex client through bili against a real upstream

This suite runs the **real `codex` CLI** against a **real Responses-compatible
upstream** through the bili proxy, and asserts the full context-management
lifecycle end-to-end:

1. **warmup** — one turn; plants a known contamination value (`1400`) that the
purity guard must later defeat.
2. **load growth** — `seq` tool outputs grow the payload deterministically
(client-side; independent of model cooperation). Asserts `[acp-usage]`
input grows.
3. **ACP compress** — under a small configured window the proxy must compress
(model-cooperative `compress requested` **or** autonomous `preflight
compressed`). Asserts usage drops after compression.
4. **purity** — post-compress recall of pre-compress facts (row counts of the
`seq` runs) must be correct and must **not** echo the planted `1400`.
5. **forge** (gated by `E2E_FORGE=1`) — establishes an ACP block under an 8k
window, then flips to a 16k window with a codex-side auto-compact limit
below the ACP trigger so codex emits a **native compaction request**;
asserts `codex compact intercepted … upstream not contacted`, a
`fc_bili_` compaction item in the codex rollout, correct echo-stripping
on the next turn, and intact recall after the forged compact.

## Running

```bash
npm run build # suite spawns dist/index.js
ACP_TEST_E2E=1 node --import tsx --test tests/e2e/e2e-codex.test.ts
```

By default the suite **skips** (`set ACP_TEST_E2E=1`) so `npm test` stays free.

## Environment

| Variable | Default | Purpose |
|---|---|---|
| `ACP_TEST_E2E` | – | `1` enables the suite |
| `E2E_UPSTREAM_URL` | `http://127.0.0.1:8199/v1` | Responses-compatible upstream (no `/bili/` prefix) |
| `E2E_UPSTREAM_KEY` | `bili-local-test` | API key passed via codex `env_key` |
| `E2E_MODEL` | `qwen3.8-27b` | model name |
| `E2E_BILI_DIST` | repo `dist/index.js` | proxy entry under test — point at any build for capability-matrix runs |
| `E2E_CODEX_BIN` | `codex` | codex binary |
| `E2E_FORGE` | – | `1` runs the forge phase (needs a dist with `BILI_CODEX_COMPACT` support, i.e. #325+) |
| `E2E_TMO` | `300000` | per-turn timeout ms |

Preflight (no tokens): `E2E_CHECK=1 node --import tsx --test tests/e2e/e2e-codex.test.ts`
prints codex version, dist path, and an upstream `/models` probe.

## Mechanics

- Full isolation: `CODEX_HOME`, `XDG_{CONFIG,CACHE,STATE}_HOME` and the work
dir (`tmp/e2e-codex-*` in the repo — **not** `/tmp`, codex refuses TMPDIR
homes) are throwaway.
- The bili window is forced via the `BILI_LAUNCHER_MODEL_WINDOWS` env
(per-model override; wins over registry peek) so compression triggers
deterministically on every run, independent of the upstream's advertised
window and of which window-alignment PRs are merged.
- Provider `name = "OpenAI"` in `config.toml` keeps codex on the remote
compaction path (V2) so the forge phase exercises the real interception.
- Assertions scrape bili's own log lines (`[acp-usage]`, `preflight
compressed`, `codex compact intercepted`, `stripped … bili compaction
item(s)`) plus codex exit codes, `--output-last-message` answers, and the
rollout JSONL.

## CI

`.github/workflows/ci-e2e.yml` runs the suite on `workflow_dispatch` with
`E2E_UPSTREAM_URL` / `E2E_UPSTREAM_KEY` from repository secrets, against the
repo's own `dist` (i.e. whatever is on master at dispatch time). The forge
phase is enabled via a repository variable `E2E_FORGE` so it can be turned on
once interception ships.
Loading
Loading