Skip to content

feat(l1): add auditable replay for historical L0 rows (#541) - #650

Open
RerankerGuo wants to merge 1 commit into
TencentCloud:mainfrom
RerankerGuo:feat/issue-541-l1-replay
Open

feat(l1): add auditable replay for historical L0 rows (#541)#650
RerankerGuo wants to merge 1 commit into
TencentCloud:mainfrom
RerankerGuo:feat/issue-541-l1-replay

Conversation

@RerankerGuo

Copy link
Copy Markdown

Summary

Closes #541.

Adds a supported Gateway admin API for replaying historical L0 rows that were skipped after an earlier L1 extraction failure and are now behind last_l1_cursor.

What changed

  • Add POST /admin/replay-l1 with required session_key and optional inclusive from, to, limit, and dry_run controls.
  • Query existing L0 rows directly from SQLite or Tencent Cloud VectorDB; no duplicate L0 seed/import writes are created.
  • Preserve original L0 message IDs in the extraction prompt and resulting source_message_ids lineage.
  • Reuse the existing L1 extraction and dedup pipeline.
  • Persist audit receipts to .metadata/l1-replay-receipts.jsonl for completed, failed, and idempotently skipped replay attempts.
  • Fingerprint exact completed batches so repeating the same replay does not invoke the LLM or write L1 again.
  • Keep the live last_l1_cursor unchanged. Historical replay uses an independent receipt ledger rather than moving the online scheduler cursor backward and accidentally reprocessing unrelated rows.

API example

curl -X POST http://127.0.0.1:8420/admin/replay-l1 \
  -H "Authorization: Bearer $TDAI_GATEWAY_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "session_key": "session-retry",
    "from": "2026-07-01T00:00:00Z",
    "to": "2026-07-02T00:00:00Z",
    "limit": 100,
    "dry_run": true
  }'

The admin route is unavailable when TDAI_GATEWAY_API_KEY is not configured; this prevents an unauthenticated caller from triggering LLM calls and persistent memory writes.

Design decisions

  • No checkpoint rewind: replaying rows behind the cursor must not reset the live scheduler cursor. The response and receipt include checkpoint_cursor_before and checkpoint_cursor_after so operators can verify this invariant.
  • Bounded execution: limit defaults to 100 and is capped at 500.
  • Range correctness: both storage backends apply session and recorded-at bounds before the limit.
  • Fail visibly: unsuccessful L1 extraction records a failed receipt and returns an error instead of being treated as a successful zero-memory replay.

Verification

  • npm test — 70/70 tests passed
  • npm run build:plugin
  • git diff --check

Focused replay tests cover:

  • dry-run range planning with no LLM call or receipt write;
  • successful extraction with original L0 IDs and unchanged checkpoint;
  • exact replay idempotency;
  • failed extraction receipt with unchanged checkpoint.

Impact

  • Breaking change: no
  • Existing capture, extraction, seed, and scheduler behavior: unchanged
  • New persistent data: append-only replay receipt JSONL under .metadata/

Add an authenticated Gateway admin endpoint for bounded L1 replay with dry-run support, original L0 lineage, exact-batch idempotency, and persisted replay receipts. Query SQLite and TCVDB directly by session and recorded-at range while leaving the live L1 checkpoint cursor unchanged.

Test: npm test (70 passed)\nTest: npm run build:plugin\nTest: git diff --check
@Maxwell-Code07

Copy link
Copy Markdown
Collaborator

Thank you for your interest and contributions! We will review your code and get back to you as soon as possible!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants