Skip to content

Keep control-plane state correct when a write or a restart fails - #293

Closed
Protonmatter wants to merge 1 commit into
BaaRai:mainfrom
Protonmatter:protonmatter/reliability-fixes-0d2e
Closed

Protonmatter wants to merge 1 commit into
BaaRai:mainfrom
Protonmatter:protonmatter/reliability-fixes-0d2e

Conversation

@Protonmatter

Copy link
Copy Markdown
Contributor

Queue an activity that could not be stored and tell the live stream the step failed. Rebuild spend caps from llm_usage, reject sessions issued before a password change, and wait for task goroutines on shutdown. A failed password read no longer mints a setup token, a coverage-flag error leaves tools available, and JSON bodies, login eviction, circuit-breaker writes, and MITM passthrough no longer fail open forever.

Queue an activity that could not be stored and tell the live stream the step failed. Rebuild spend caps from llm_usage, reject sessions issued before a password change, and wait for task goroutines on shutdown. A failed password read no longer mints a setup token, a coverage-flag error leaves tools available, and JSON bodies, login eviction, circuit-breaker writes, and MITM passthrough no longer fail open forever.

@BaaRai BaaRai left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work. Two blockers before any code review:

  1. The branch conflicts with main in db/migrate_test.go.
  2. 0005_activity_outbox.sql collides with 0005_traffic_rewrite_rules.sql, which is already on main. With both present, loadMigrations fails ("migration version 0005 is used by both"), and the binary cannot open any database. Please renumber the outbox migration to 0006_activity_outbox.sql and rebase on current main.

Please also split the PR: it bundles several independent changes (auth, shutdown drain and outbox, budget caps, coverage flag, llmpool write queue, MITM passthrough TTL, JSON body cap). Two questions for the budget and shutdown parts: TaskLLMSpend counts every llm_usage row while the task ledger previously counted planner rounds only, so is the turn cap meant to change after a restart? And does cancelling task contexts in Engine.Wait persist a terminal status that would stop restoreTaskRuntimes from resuming those tasks?

@Protonmatter

Copy link
Copy Markdown
Contributor Author

Rebased onto current main. 0005_activity_outbox.sql is now 0006_activity_outbox.sql, so it no longer collides with 0005_traffic_rewrite_rules.sql. db/migrate_test.go lists both.

The bundle is split:

Turn cap. The live ledger does not count planner rounds only. BudgetLedger.Record counts each planner completion, and IntentMeter.Record adds each worker completion to the same task spend (TestBudgetCountsMatchUsageFixture expects 2 planner calls + 3 worker calls = 5 turns). A round with several tool calls is several turns.

COUNT(*) over every llm_usage row was wider than that. Main-agent chat (worker=main), goal decomposition (goals), and side questions (btw) never enter the budget sink, so a restart could trip the turn cap earlier than the running process. Judge and compactor calls made from a planner or worker context do enter the sink: the judge reuses that context, and the compactor keeps it through context.WithoutCancel. #302 counts only planner, worker, judge, and compactor. A judge call made from the main agent is still tagged judge and would be included on restore. That is the remaining mismatch.

Shutdown. Engine.Wait does not write a terminal task status. The planner loop and the deadline coordinator return when their context ends, and restoreTaskRuntimes still starts every non-queued, non-terminal task.

The intent claim was the hole. runIntent did not treat AbortShutdown specially, so a cancel inside a worker could settle the intent to done, blocked, or exhausted. ResetRunningIntents only reopens running, so that intent would not come back. AbortShutdown says leftover running intents are reset to open. #300 cancels with that cause and leaves the claim running.

Closing this PR in favor of the seven above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants