Skip to content

fix(runtime): preserve prompt cache and durable provider context - #693

Merged
XuPeng-SH merged 13 commits into
matrixorigin:mainfrom
XuPeng-SH:fix/prompt-cache-dsml-harness
Sep 4, 2026
Merged

fix(runtime): preserve prompt cache and durable provider context#693
XuPeng-SH merged 13 commits into
matrixorigin:mainfrom
XuPeng-SH:fix/prompt-cache-dsml-harness

Conversation

@XuPeng-SH

@XuPeng-SH XuPeng-SH commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Sanitize degraded DSML tool-call envelopes while preserving raw parser input.
  • Model cache delivery as typed provider capabilities and preserve a byte-stable prompt prefix without provider/model-name branches.
  • Keep runtime-owned authority append-only and distinct from human user intent across canonical history, compaction, resume, UI, and Work admission.
  • Make provider-attempt admission a durable WAL boundary with strict parent/head and canonical transition checks.
  • Compare canonical append suffixes only after the same metadata-only wire projection applied to the provider request.
  • Derive cache diagnostics from the immutable prepared provider request.
  • Honor authentic live v2 whole-workspace verification receipts while keeping legacy settlement fail-closed.

Root causes and design

The previous implementation could rewrite provider-visible history between rounds, lose canonical appends across a crash, and compare provider-projected messages with a raw canonical suffix that still carried internal typed metadata. Those conflicts reduced prompt-cache reuse and could reject valid session continuation.

Separately, completion settlement first accepted a typed whole-workspace executor receipt, then rejected the same evidence through a legacy latest-single-artifact text gate. This produced a false execution_incomplete outcome even though the workspace was correctly changed and verified.

Correctness remains independent from prompt cache. Runtime provenance, durable lineage, Work ownership, tool execution, and completion authority stay typed and fail closed. The full-scope lane requires an executed successful bash invocation with live authoritative arguments, mode=verify, exact v2 source/schema/ownership, bound-workspace scope, no contradictory or later mutation, and no quarantine. Restored records deliberately cannot reuse this live authority.

There are no task-, filename-, command-, model-, prompt-, language-, or error-string special cases.

Validation

  • cargo test -p astra-runtime --lib: 5198 passed, 0 failed, 42 ignored.
  • cargo check -p astra-runtime --all-targets.
  • cargo fmt --all -- --check.
  • Focused positive and negative receipt tests, including Edge callback → production record ingestion → completion settlement.
  • DeepSeek Flash Max regression on 24412c65e: 3/3 reward 1, 0 exceptions/retries, aggregate cache rate 93.60% (prior baseline 82.80%).
  • Final 0fae5e3e2 GLM 5.2 regression: 3/3 reward 1, 0 exceptions/retries, aggregate cache rate 90.11%; all server-authoritative completed with complete tool/token ledgers.
  • The previously false-negative modernize-scientific-stack path is now reward 1 and authoritative completed after exercising the typed post-mutation verification lane.
  • Earlier branch validation: runtime nextest 5497 passed / 147 skipped; make check passed.

Raw benchmark evidence is retained locally under /home/xupeng/bench_result_0901/regression-24412c and /home/xupeng/bench_result_0901/regression-0fae5e3.

@XuPeng-SH XuPeng-SH mentioned this pull request Sep 3, 2026
10 tasks
XuPeng-SH added a commit that referenced this pull request Sep 3, 2026
## Summary

External fork `pull_request` runs are not admitted into workflow
concurrency until a maintainer approves them. During that approval
window, `cancel-in-progress: true` cannot supersede the previous head,
so stale CI keeps consuming runners.

This change closes that gap without giving fork code additional
authority:

- add a fork-only `pull_request_target` controller that checks out the
exact trusted `github.workflow_sha` and only calls the Actions API;
- use the synchronize payload's immutable `before` SHA as the sole
cancellable generation, so an old controller can never cancel its
`after` head or a later push;
- intentionally do not coalesce controller events: every `before ->
after` transition must cancel its own prior generation during rapid
pushes;
- query workflow runs directly by `head_sha`, then require the exact
source-repository ID and source ref because GitHub currently returns an
empty `pull_requests` array for these fork runs;
- replace job-level `always()` with `!cancelled()` in Test Suite and
Static Checks, preserving fail-safe execution after scope-classifier
failure while allowing a superseded workflow to terminate;
- key normal workflow concurrency by PR number rather than branch name,
preventing identically named branches in different forks or PRs from
cancelling each other;
- bound pagination, request duration, cancellation fanout, and HTTP 409
polling; reject redirects, cleartext API roots, cross-origin pagination,
and false-success handling of an active conflict;
- isolate per-run cleanup failures so later stale runs are still
cancelled, then fail the controller with an aggregate diagnostic;
- run every `scripts/ci/test_*.py` contract in Static Checks and
document the behavior.

## Related issue

N/A. The failure was observed directly on Astra PR #693. Run
`33779474796` started at 16:34 UTC and continued until 16:51; its online
and CLI jobs substantially completed before the run became cancelled.
The replacement run started its jobs at 16:51. This also exposed that
job-level `always()` resisted cancellation.

## Change type

- [ ] Feature
- [x] Bug fix
- [x] Documentation
- [x] Refactor or performance improvement
- [x] Test
- [x] Build, CI, or maintenance

## User and compatibility impact

Fork PR updates stop spending runner time on superseded heads without
waiting for replacement-run approval. Current and later heads are never
cancellation candidates. Internal PRs incur no controller job. No
product API, configuration, or runtime behavior changes.

## Architecture and complexity delta

- Canonical owner changed or extended: the trusted workflow revision
owns pre-admission fork cancellation; ordinary per-workflow concurrency
remains the admitted-run fast path.
- Existing implementations and callers searched: Astra Test Suite,
Static Checks, PR Title concurrency, CI routing/contracts, GitHub
synchronize payload and Actions API contracts, and MatrixOne's
`pull_request_target`/`!cancelled()` patterns.
- Superseded code, states, tables, shims, or self-only tests removed:
branch-name-only concurrency identity, non-current-head scanning,
live-head time-of-check guard, and job-level cancellation-resistant
`always()` conditions.
- If parallel implementations remain, their boundary and retirement
condition: normal concurrency handles admitted runs; the controller
handles the external-fork pre-admission gap. Both converge through
GitHub's idempotent cancellation API.

Security boundary: the controller has only `actions: write` and
`contents: read`. It checks out exactly `github.workflow_sha` with
credential persistence disabled, never checks out or executes the PR
head, uses no repository secrets, sends its token only to the configured
HTTPS API origin, and rejects redirects.

Generation model: each synchronize event owns only its exact
GitHub-provided `before` SHA. Controllers are not coalesced, so
processing any permutation of rapid updates cancels every prior
generation while making the latest generation unreachable to older
controllers.

## Verification

- Commands and results:
- `python3 -m unittest discover -s scripts/ci -p 'test_*.py'` — PASS, 38
tests selected.
- `python3 -m py_compile scripts/ci/cancel_stale_pr_runs.py
scripts/ci/test_cancel_stale_pr_runs.py` — PASS.
  - `python3 scripts/ci/validate_repository.py` — PASS.
- `go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12
.github/workflows/*.yml` — PASS.
- `git diff --check origin/main...HEAD` — PASS after rebase onto
`87856b74e`.
- Read-only live API query by old fork head SHA returned the expected
Test Suite, Static Checks, and PR Title runs with stable
`head_repository.id`, `head_branch`, and `head_sha` fields.
- The final selector was exercised read-only against PR #694's live API
payload and selected only its exact active Test Suite generation.
- Public entrypoint exercised: local controller contracts cover the
event/API boundary. GitHub loads a new `pull_request_target` workflow
only from the default branch, so live acceptance begins after merge.
- Unhappy paths exercised: all controller execution orders for rapid
pushes; current/newer generations; same ref in another fork; another ref
in the same fork; all active and completed statuses; duplicate
pagination observations; pagination cycles; completion racing
cancellation; concurrent double-cancellation with delayed terminal
state; persistently active HTTP 409; one cancellation failure followed
by successful cleanup of later runs; cross-origin redirect and HTTP
downgrade; option-shaped Git ref; oversized cancellation fanout;
repository-wide PR-workflow count exceeding the bounded fanout; exact
trusted workflow checkout; main/develop co-location through
`github.workflow_sha`; cancellation-resistant job conditions.
- Database verification: N/A; no database or runtime path changes.

Post-merge live acceptance plan:

1. Start a fork PR run, then push a second head without approving the
replacement run.
2. Verify `Supersede stale PR runs` starts without external-contributor
approval and cancels the exact prior Test Suite/Static Checks runs while
preserving the new pending head.
3. Push three heads rapidly and verify every controller runs, all three
prior generations stop, and the latest head remains untouched.
4. Verify cancellation actually terminates Test Suite jobs rather than
allowing job-level `always()` work to complete.
5. Verify two PRs using the same branch name have distinct normal
concurrency groups.
6. Repeat with a PR targeting `develop` and verify the controller
executes the script co-located with its trusted workflow revision.

## Final checklist

- [x] I added or updated tests at the layer that owns the behavior, or
explained why no test is needed.
- [x] I updated public or design documentation for contract changes, or
the change needs no documentation update.
- [x] I checked the diff for credentials, private URLs, customer data,
generated files, and other sensitive information.
- [x] The PR title follows the repository's Conventional Commit format.
@XuPeng-SH XuPeng-SH changed the title fix(runtime): stabilize prompt cache and sanitize tool streams fix(runtime): preserve prompt cache and durable provider context Sep 4, 2026
@XuPeng-SH
XuPeng-SH force-pushed the fix/prompt-cache-dsml-harness branch from 0fae5e3 to 1670a36 Compare September 4, 2026 04:15
@XuPeng-SH
XuPeng-SH force-pushed the fix/prompt-cache-dsml-harness branch from 609a55d to 7920d5f Compare September 4, 2026 04:59
@XuPeng-SH
XuPeng-SH merged commit 23ee674 into matrixorigin:main Sep 4, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant