perf(runtime): reduce database-bound turn latency

[gouhongshen]

Summary

Ports the database-bound turn-latency work from #656 onto current main without restoring implementation that main has already superseded.

High MatrixOne latency magnifies every serialized read and write around a warm agent turn. This change keeps the same durable and authorization boundaries while reducing avoidable round trips and overlapping work that is causally independent:

• atomically acquires the canonical writer and reserves the next turn, and loads the session admission snapshot once;
• runs distributed admission, immutable history materialization, and canonical authority admission concurrently, with explicit release on every partial-success path;
• captures Tool/Skill discovery once per request and reuses the frozen snapshot while building runtime capability projections;
• batches messaging claims/releases and trace-event inserts instead of opening one transaction per item;
• reuses already-loaded run, quota, provider-attempt, invocation, and task-board facts inside the same lifecycle boundary;
• atomically settles the successful provider attempt, request context, and logical invocation;
• keeps the terminal response critical path limited to canonical commit, provider settlement, and the fenced terminal transition, then runs derived projections after terminal SSE closes and only for the owner that committed the terminal;
• keeps lightweight reasoning_done / thinking_done markers durable for reconnect while raw reasoning content and deltas remain live-only;
• scopes compiled prompt static-section caching to PipelineSession;
• separates canonical manifest-chain and materialized-content hash domains during compaction;\n- reactivates fully verified legacy reachable = 0 staged manifests inside the canonical commit transaction, preserving generation and aggregate fences without adding a read to the new-manifest hot path.

The branch was rebuilt directly on current main. It contains no #663 Edge registry commit or implementation.

Main-specific adaptation is deliberate:

• fix(runtime): preserve prompt cache and durable provider context #693 preserves final wire-prompt cache diagnostics and durable provider-context recovery; the rebase keeps those semantics while retaining this PR's session-scoped static-section cache.

• feat: prepare Astra for open source release #639 removed the legacy MatrixOne task-board owner; this PR does not reintroduce it.

• main already owns atomic composite checkpoint persistence; this PR keeps that write-through contract and does not restore the rejected heavy-checkpoint deferral state.

• main performs authoritative logical-attempt recovery before prompt capture. Prompt capture remains after durable admission so a recovered attempt cannot persist a prompt under the preceding attempt identity.

No heuristic fast path, Tool/Skill suppression, authorization relaxation, schema migration, or compatibility fallback is introduced.

Related issue

Follow-up/main port of #656. The original latency investigation and paired MOI integration are tracked in matrixorigin/matrixflow#16389.

Change type

• Feature
• Bug fix
• Documentation
• Refactor or performance improvement
• Test
• Build, CI, or maintenance

User and compatibility impact

No intended user-visible or API compatibility change. Warm Server turns perform fewer serialized database operations. Terminal SSE is published immediately after the durable correctness boundary; slower derived projections cannot extend the visible stream tail. Reconnect still receives reasoning/thinking completion markers. No configuration, schema, or migration is required.

Architecture and complexity delta

• Canonical owner changed or extended: SessionContextCoordinator owns the combined admission snapshot and atomic writer-plus-turn reservation; inference execution owns atomic successful settlement; runtime lifecycle owns request-scoped capability discovery reuse and owner-fenced post-terminal projection ordering; PipelineSession owns compiled static prompt sections.
• Existing implementations and callers searched: Server run admission/lifecycle, canonical session coordinator and commit path, inference invocation/provider-attempt settlement, runtime capability discovery, messaging transport, trace persistence, resource governor, storage readiness, prompt construction, finalization, compaction, reconnect replay, and CLI state construction.
• Superseded code, states, tables, shims, or self-only tests removed: repeated preflight reads and serialized independent writes are replaced in place. The legacy task-board MatrixOne owner and checkpoint-deferral state removed from main remain removed. The previously stacked fix(edge): make registry claim settlement idempotent #663 Edge registry changes were removed from this branch entirely. No compatibility shim or parallel store is added.
• If parallel implementations remain, their boundary and retirement condition: none for the changed behavior. The optional frozen discovery snapshot is request-scoped evidence consumed by the existing capability owner, not a second capability authority.

Net diff against current main: 25 files, +2566/-858; zero database tables, migrations, public statuses, or configuration fields.

Verification

• Commands and results:
  • cargo fmt --all -- --check — passed on the rebuilt branch.\n - cargo check -p astra-services --tests — passed after the staged-manifest compatibility fix.\n - cargo clippy -p astra-services --tests -- -D warnings — passed after the staged-manifest compatibility fix.\n - cargo test -p astra-services session_context --lib — passed; 1,861 unrelated tests filtered by the targeted name.
  • cargo check -p astra-services -p astra-runtime — passed before the review-only terminal ordering correction; its affected runtime target was compiled again by the tests below.
  • cargo clippy -p astra-services -p astra-messaging -p astra-turn-core -p astra-runtime --all-targets -- -D warnings — passed before the review-only terminal ordering correction.
  • cargo test -p astra-runtime superseded_terminal_skips_all_owner_derived_projections -- --nocapture — passed; a Superseded owner gate invokes none of the derived projection sinks.
  • cargo test -p astra-runtime terminal_events_for_persistence_keeps_only_terminal_lifecycle_events -- --nocapture — passed; raw reasoning remains excluded while completion markers persist.
  • cargo test -p astra-runtime turn::llm::durable::tests --lib -- --nocapture — 48 passed.
  • cargo test -p astra-runtime turn::canonical_commit::tests --lib -- --nocapture — 2 passed after the fix(runtime): preserve prompt cache and durable provider context #693 rebase; provider replacement remains bound to the materialized conversation root while commit fencing retains the separate manifest root.
  • cargo test -p astra-runtime typed_objective_relations_survive_real_tiered_compaction --lib -- --nocapture — passed after migrating the fix(runtime): preserve compacted user anchor at canonical commit #690 regression to from_materialized_admission.
  • cargo clippy -p astra-runtime --all-targets -- -D warnings — passed after resolving the fix(runtime): preserve prompt cache and durable provider context #693 conflict and canonical-proof integration.
  • cargo test -p astra-services resource_governor --lib — 17 passed.
  • cargo test -p astra-turn-core static_sections_cache_is_scoped_to_pipeline_session — passed before the main rebase; the post-rebase workspace compile and Clippy gates cover the adapted call sites.
  • cargo test -p astra-runtime compaction_ --lib — 114 passed before the main rebase; the main port retains the same compaction commits and post-rebase compile/Clippy gates passed.
  • make test-contract — 10 admin, 4 HTTP, and 2 settings contract tests passed.
  • make test-offline — progressed through Server-only runtime and Web profile tests, then stopped in an unchanged astra-edge test because this macOS host rejected its temporary workspace-lock namespace. This branch changes neither crates/astra-edge nor crates/astra-tools; the same host condition is independent of this diff.
  • live MatrixOne completion_http_boundary_persists_success_failure_and_caller_cancellation — passed twice on this exact port; 2.29s including first schema readiness and 0.42s warm.
  • git diff --check origin/main...HEAD — passed.
  • live MatrixOne cancellation test after synchronizing current main — passed on macOS and Linux; Linux completed in 2.46s.
  • the cancellation integration test reports invocation, provider-attempt, and settlement-debt facts if CI convergence times out, so any platform-only failure is diagnosable rather than opaque.
  • reproduced the cancellation failure against CI-equivalent matrixone:latest: INSERT IGNORE reported one affected row for a duplicate debt, so the unsafe affected-row early return skipped authoritative identity recovery; removing that early return restored exact provider_attempt_id recovery, and the formerly deterministic failure passed 20/20 consecutive runs.
• Public entrypoint exercised: the original perf(runtime): reduce database-bound turn latency #656 tree was exercised through local MOI /chat/stream with Astra Server and EPH Sandbox, including warm follow-up turns under injected SQL latency. This exact main port was not redeployed.
• Unhappy paths exercised: authority/cursor conflicts, partial admission cleanup, cancellation and lease loss, provider delivery uncertainty, terminal CAS supersession, settlement failure/recovery, missing discovery snapshots, compaction hash mismatch, and quota boundaries.
• Database verification: perf(runtime): reduce database-bound turn latency #656 was exercised against local MatrixOne, including ignored MatrixOne persistence tests. This port adds an ignored MatrixOne regression that pre-stages the exact legacy manifest, segment, and reference with reachable = 0, then proves retry activation and materialization. This main port adds no schema or migration. Its caller-cancellation settlement path was rerun against local MatrixOne twice and converged correctly; the remaining adapted SQL paths retain the same transactional/fencing contracts and are covered by compile, Clippy, durable lifecycle, and contract tests above.

Final checklist

• I added or updated tests at the layer that owns the behavior, or explained why no test is needed.
• I updated public or design documentation for contract changes, or the change needs no documentation update.
• I checked the diff for credentials, private URLs, customer data, generated files, and other sensitive information.
• The PR title follows the repository's Conventional Commit format.

[XuPeng-SH]

结论：当前不建议合并，请先修复以下问题。

1. [P1] 流式执行失去 terminal ownership 后仍会写派生状态

crates/runtime/src/server/run/lifecycle/mod.rs 的 terminal projection block（约 L16629-L16684）没有受 owner_terminal_committed 约束，并调用了无 generation fence 的 persist_usage、transcript materialization 和 run_after_core。当 terminal CAS 返回 Superseded、authority 不确定或 generation 过期时，旧 executor 仍可能覆盖 usage、写入 CSL/hook/observer/promotion/projection，污染企业审计、计费和 ContextPipeline 数据。

请让所有派生写入重新受 terminal owner fence 保护；删除额外的 unfenced usage 写入，只保留 atomic settlement 或 persist_usage_if_current_owner。需要补 terminal CAS=Superseded 的测试，断言这些派生 sink 不会被调用。

2. [P1] best-effort projection 被放入 terminal SSE 关键路径

同一段代码在发布 terminal SSE 前 tokio::join! 等待 usage、transcript、hook DB、Memoria observer、SessionEnd hooks、promotion 和 projection。任一 sink 变慢都会阻塞已经 durable committed 的回答，导致客户端超时并触发重复请求；这也与本 PR 的降低 turn latency 目标相悖。

canonical commit、provider settlement、terminal CAS 才是响应正确性的关键路径。请恢复“先完成 terminal SSE，再执行派生 projection”的边界；如果必须提前执行，也必须有明确超时、降级和 owner fence。

3. [P2] 删除 reasoning_done / thinking_done 的 durable replay 会破坏断线恢复

crates/runtime/src/server/run/lifecycle/run_state.rs 的 terminal persistence 和 live persistence 过滤掉了 reasoning/thinking completion marker。客户端已经收到 reasoning delta 后断线重连时，没有 completion marker 可能无法关闭 thinking UI，直到整个 run 结束。

请保留轻量 completion marker，或先定义并实现明确的 reconnect state snapshot，再删除 marker，并补充断线恢复测试。

4. [P1/process] PR 混入了尚未批准且已过时的 #663

PR #670 的 commits 包含 #663 的旧版 edge registry commit（9728d51），但 #663 仍为 Open / CHANGES_REQUESTED，当前 head 已经不同；当前 main 也不包含该 commit。PR 描述中“#663 changes are already part of the base”不成立。请基于最新 main 重新整理纯 latency 分支，避免把未批准的 edge claim/heartbeat/unregister 实现带入本 PR。

正向评价：atomic writer+turn admission、successful inference settlement、PipelineSession-scoped prompt cache、request-scoped discovery snapshot，以及 hash-domain separation 的方向是正确的，也符合 Astra 的企业 durable-agent 定位。但性能优化必须服从 authority fencing、Trace/审计完整性、Runner/Edge side-effect recovery 和断线恢复语义。

[gouhongshen]

已按 review 完成自检和修复，提交为 6abe5e99：

1. 删除了流式终态前无 generation fence 的 persist_usage，保留 atomic settlement / persist_usage_if_current_owner；journal、CSL/transcript、hooks/observer/promotion、Work/Memory/workspace 等派生副作用统一受 owner_terminal_committed gate 约束。新增 Superseded 回归测试，确认 gate 内的派生 sinks 均不执行。
2. 派生 projection 已移到 drop(event_tx) 之后；terminal SSE 不再等待这些 best-effort 写入。
3. 恢复 reasoning_done / thinking_done 的轻量 durable marker；raw reasoning content/delta 仍保持 live-only。
4. 分支已从最新 origin/main 重建并 force-with-lease 更新；9728d51、相关 merge commits，以及 edge_registry.rs / edge_dispatch_db_it.rs diff 均已移除。PR 描述中的错误 base 声明和净 diff 统计也已更正。

本地定向验证通过：两项新增/恢复的 runtime tests、cargo fmt --check、git diff --check。新的 CI 已由重写后的纯 latency 分支触发。

[gouhongshen]

The current head 6abe5e99 addresses all four findings from the review of 9c48ca57:

1. All journal, transcript, hook, observer, promotion, projection, and workspace-cleanup writes are now inside run_owner_fenced_terminal_projections(owner_terminal_committed, ...). A Superseded terminal owner invokes none of those sinks; the new deterministic test verifies zero calls.
2. The terminal events and turn_complete are sent and event_tx is dropped before derived projections run. The only non-atomic usage repair left in the durable boundary is persist_usage_if_current_owner; the unfenced usage write is gone.
3. reasoning_done and thinking_done are again retained as lightweight durable terminal/reconnect markers, while raw reasoning/thinking deltas remain excluded. The persistence test covers both markers and raw-content exclusion.
4. The branch was rebuilt directly on current main; git diff origin/main...HEAD contains no edge_registry.rs or edge_dispatch_db_it.rs, so no fix(edge): make registry claim settlement idempotent #663 implementation remains in perf(runtime): reduce database-bound turn latency #670.

The PR description has also been updated to reflect these ownership, SSE ordering, reconnect, and branch-scope corrections. Please re-review the current head rather than the superseded reviewed commit.

[XuPeng-SH]

当前 head 已修复上轮四个问题，但仍有一个需要阻断合并的持久化兼容性问题：

[P1] 复用历史 staged manifest 时没有把它重新激活

persist_database_immutables_in_tx 现在只在新插入时把 reachable 写成 1；当 INSERT IGNORE 命中已存在的相同 manifest 时，代码仅校验 manifest_json，不会把已有行从 reachable = 0 更新为 1。见：

Astra/crates/services/src/session_context_coordinator.rs

Lines 2597 to 2660 in 6abe5e9

	let manifest_insert_sql = matrixone_statement_with_null_shape(
	"INSERT IGNORE INTO conversation_manifest_nodes
	(isolation_domain, owner_user_id, session_id, branch_id, manifest_root,
	parent_manifest_root, completed_turn, conversation_seq,
	compaction_generation, canonical_segment_bytes, total_canonical_bytes,
	total_message_count, manifest_json, reachable)
	VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)",
	[node.parent_manifest_root.is_some()],
	);
	let result = sqlx::query(&manifest_insert_sql)
	.bind(&key.isolation_domain)
	.bind(&key.owner_user_id)
	.bind(&key.session_id)
	.bind(&key.branch_id)
	.bind(&node.manifest_root)
	.bind(&node.parent_manifest_root)
	.bind(i64::from(node.completed_turn))
	.bind(i64_from_u64(
	"conversation sequence",
	node.conversation_seq,
	)?)
	.bind(i64_from_u64(
	"manifest compaction generation",
	node.compaction_generation,
	)?)
	.bind(i64_from_u64(
	"manifest segment bytes",
	canonical_segment_bytes,
	)?)
	.bind(i64_from_u64(
	"manifest total canonical bytes",
	total_canonical_bytes,
	)?)
	.bind(i64_from_u64(
	"manifest total message count",
	total_message_count,
	)?)
	.bind(database_to_json("manifest", node)?)
	.execute(&mut **tx)
	.await
	.map_err(|source| database_error("persist_manifest", source))?;
	if result.rows_affected() == 0 {
	let stored = sqlx::query(
	"SELECT manifest_json FROM conversation_manifest_nodes
	WHERE isolation_domain = ? AND owner_user_id = ?
	AND session_id = ? AND branch_id = ? AND manifest_root = ?",
	)
	.bind(&key.isolation_domain)
	.bind(&key.owner_user_id)
	.bind(&key.session_id)
	.bind(&key.branch_id)
	.bind(&node.manifest_root)
	.fetch_one(&mut **tx)
	.await
	.map_err(|source| database_error("verify_existing_manifest", source))?
	.try_get::<String, _>("manifest_json")
	.map_err(|source| database_error("decode_existing_manifest", source))?;
	let stored: ContextManifestNodeV1 = database_json("existing_manifest", &stored)?;
	if stored != *node {
	return Err(SessionContextCoordinatorError::NeedsRepair(
	"existing immutable manifest does not match its content-addressed key".into(),
	));
	}
	}

这是升级兼容场景，不是理论分支：当前 main 的旧流程会先在独立事务里写入默认 reachable = 0 的 manifest，再在 canonical commit 事务里执行 UPDATE ... SET reachable = 1。进程中断、cursor conflict 或 commit 失败都可能留下内容完整但不可达的 staged manifest。升级到本 PR 后重试相同确定性 delta 会命中这行；随后 session head 被提交为该 manifest root，但 materialize 仍按 reachable = 1 过滤，canonical head 因而指向不可物化的历史。

请在确认 existing manifest 及其 references 完全匹配后，于同一事务显式激活该行（保留 generation/totals 等 fence，并校验恰好一行），并增加回归测试：预置相同 root、reachable = 0 的旧版 staged manifest，重试 commit 后应变为 reachable 且可 materialize。

[gouhongshen]

已修复这个 P1，并将分支 rebase 到最新 main。修复在 7ca196d：新 manifest 仍走一次直接 INSERT 的热路径；只有命中历史重复主键时才进入兼容慢路径，在同一事务中 FOR UPDATE 校验完整 manifest、独立 totals/generation/segment-byte 字段和有序 references。仅当已验证旧行的 reachable = 0 时，才以 generation/totals 等 fence 更新为 1，并要求恰好更新一行；已可达的同内容行保持幂等。新增 MatrixOne 回归用例预置旧版完整 staged manifest（reachable = 0），验证 retry 后可达且能够 materialize。cargo check -p astra-services --tests、cargo clippy -p astra-services --tests -- -D warnings 和格式检查已通过。PR 描述也已同步。

[XuPeng-SH]

重新审查了最新 head 7ca196d3。上轮指出的 legacy staged manifest 问题已经正确修复：existing row 与 references 先被完整校验，reachable = 0 再在同一事务内带 fence 激活，并有真实 MatrixOne 的 materialize 回归测试。

当前仍有一个合并阻断项：

[P1] rebase 后 runtime 测试无法编译

最新 main/#690 增加的测试仍调用已被本 PR 重命名移除的 CanonicalRewriteProof::new：

Astra/crates/runtime/src/server/run/lifecycle/tests.rs

Lines 833 to 837 in 7ca196d

	// Admit an actual prefix, then prove the real compaction rewrites it.
	let prior = messages[..7].to_vec();
	let root = astra_turn_types::canonical_conversation_root(&prior);
	let mut proof = CanonicalRewriteProof::new(&prior, &root, 0);
	let permit = proof.begin(&messages);

本 PR 只保留 CanonicalRewriteProof::from_materialized_admission，因此 clippy、astra-runtime、core crates 和 online integration 都在同一处报 E0599；这不是 CI/flaky 或网络问题，而是确定性的 source integration failure。

请把该测试迁移到 from_materialized_admission，并继续保持两个 hash domain 分离：第二个参数应表达 synthetic/admitted manifest root，不要重新把 canonical_conversation_root(&prior) 当成 manifest root。修复后至少确保 runtime test compile、clippy 和相关测试通过。

[gouhongshen]

已修复最新 P1，并同时处理了冲突和同源编译问题。分支已 rebase 到 23ee6747（#693），冲突合并保留了 #693 的 final wire-prompt cache diagnostics / durable provider-context recovery，也保留本 PR 的 session-scoped static-section cache。a906bc44 将 #690 lifecycle 测试迁移到 CanonicalRewriteProof::from_materialized_admission，传入独立的 synthetic manifest root，并显式断言它与 materialized conversation root 不同。自检还发现 canonical commit 自身有 4 处旧构造器和 2 处旧字段引用，因此一并迁移，并新增独立 base_prefix_root 保存 durable conversation hash；base_manifest_root 继续只承担 commit fence，两个 hash domain 没有重新混合。验证通过：runtime 全测试目标编译、2 个 canonical proof tests、typed-objective real-compaction regression、cargo clippy -p astra-runtime --all-targets -- -D warnings、格式和 diff check。PR 描述已同步。