feat: v3.1.0 — lazy conversation labeling, off process shutdown (#65) - #66
Merged
Conversation
…dzs) Adversarial review of radiantlogicinc#65 and its eleven findings. Shutdown no longer calls LLM_CONVERSATION_STORE; conversations are labeled when they are used as a chat. The harm was not a missing title. finalize_conversations_on_shutdown ran up to max_live_sessions (50) sequential, untimed LLM calls, and it ran BEFORE stop_all_chat_sessions writes checkpoints — so under a 30s termination grace period the process was killed generating titles and lost everything each channel had accumulated since its last eviction. Shutdown labeling was also already LRU roulette: the ~92% of conversations evicted before SIGTERM were never labeled at all. - Shutdown makes unsaved turns durable through the same save_conversation_ incremental chokepoint the turn path uses, and leaves blank placeholders. The active_conversation_id == 0 fallback is kept: it is the only path that records turns for a channel whose persist was skipped by a raising work_fn. - Labels fill on the first completed chat turn (after DONE, so the LLM call is outside the drain, off the turn's critical path and outside the 409 window), on activate when still blank, and on rotate. Refreshed at turn counts 1, 4, 16, 64, … so a long thread costs O(log n) calls instead of one or n; labeling once and never again would give a 40-turn thread the title of its first exchange. - Generation is bounded at the LLM client, not by the caller's await: timeout and num_retries=1 go to dspy.LM. asyncio.wait_for would cancel the wait while the thread ran on, and CPython then joins default-executor threads for up to THREAD_JOIN_TIMEOUT (300s) at exit. New LLM_CONVERSATION_STORE_TIMEOUT_SECONDS (default 12), resolved process-environment-first so a container override is reachable, validated and logged at startup. - "conversation summary" now identifies its turn on the deterministic and action paths. Both hardcoded a constant ("assistant_mode_command", "process_action command"), and since generate_topic_and_summary reads only that field, /initialize conversations were asking an LLM to title a fixed string. Bounded and newline-collapsed; parameters stay in conversation_traces. - Blank topic is now a sound "no title yet" sentinel. An empty generated topic collided with every other unlabeled conversation and stored as ' 1', which counted as labeled forever. - ConversationStore._ensure_unique_topic excludes the record being written; a rewrite with an unchanged topic oscillated between 'T' and 'T 1'. - New get_conversation_window: the restore paths kept 20 turns but READ every turn, so at 450 KB payloads restoring a long conversation was tens of MB resident. - /new_conversation persists before labeling. In the reserved-but-unwritten state it spent a generation and then raised out of update_conversation_topic_summary, making the rotate that would have cleaned the state up the one unavailable operation. - Streaming EOF no longer waits on the label (new on_done on run_owned_turn). invoke_agent, the MCP-exposed tool, IS the streaming endpoint, and fastapi-mcp reads the whole stream as the tool result. Three of radiantlogicinc#65's specific designs were wrong and are corrected here: preview rested on that constant-literal field, its per-call timeout could not bound the process, and its call site would have put the LLM call back inside the drain it was written to protect. Full findings and resolutions in bd epic fix-dzs (11 children). Verified: full suite 1788 passed, 15 skipped, 0 failed (48:56); trained example artifacts byte-identical apart from regenerable build caches. Four fixes verified by falsification — reverting each makes its own test fail and no other. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Sorry @dharrawal, your pull request is larger than the review limit of 150000 diff characters
Reviewer's GuideImplements lazy, bounded conversation labeling and removes LLM work from shutdown, adds windowed conversation reads, tightens topic uniqueness/blank handling, and wires labeling into turn and streaming lifecycles with new configuration and tests. Sequence diagram for lazy conversation labeling after a completed turnsequenceDiagram
actor User
participant FastAPIRoute as invoke_agent
participant Turns as run_owned_turn
participant Utils as save_conversation_incremental
participant Label as _label_conversation_after_turn
participant LabelHelper as try_ensure_topic_and_summary
participant StoreHelper as ensure_topic_and_summary
participant Store as ConversationStore
participant LLM as generate_topic_and_summary
User->>FastAPIRoute: POST /invoke_agent (chat turn)
FastAPIRoute->>Turns: run_owned_turn(runtime, turn_registry, execn, work, session_manager, on_done)
Turns->>Utils: save_conversation_incremental(runtime, extract_turns_from_history, logger)
Utils-->>Turns: turns_appended
Turns->>Turns: execn.exec_state = DONE
Turns->>Label: _label_conversation_after_turn(runtime, execn, turns_appended)
Label->>LabelHelper: try_ensure_topic_and_summary(runtime, generate_topic_and_summary, turns_appended)
LabelHelper->>StoreHelper: ensure_topic_and_summary(runtime, generate_topic_and_summary, turns_appended)
StoreHelper->>Store: get_conversation_label_state(conversation_id)
Store-->>StoreHelper: (stored_topic, durable_turns)
StoreHelper->>StoreHelper: _label_is_due(stored_topic, durable_turns, turns_appended)
alt label due
StoreHelper->>Store: get_conversation_summaries(conversation_id)
Store-->>StoreHelper: turns
StoreHelper->>LLM: generate_topic_and_summary(turns)
LLM-->>StoreHelper: (topic, summary)
StoreHelper->>Store: update_conversation_topic_summary(conversation_id, topic, summary)
else label not due or blank topic
StoreHelper-->>LabelHelper: None
end
LabelHelper-->>Label: (topic, summary) or None
Label-->>Turns: labeling complete
Turns-->>FastAPIRoute: turn finished (response already sent via on_done)
Sequence diagram for activation with windowed restore and lazy labelingsequenceDiagram
actor User
participant FastAPIRoute as activate_conversation
participant Runtime as ChannelRuntime
participant Store as ConversationStore
participant Utils as restore_history_from_turns
participant LabelHelper as try_ensure_topic_and_summary
participant StoreHelper as ensure_topic_and_summary
participant LLM as generate_topic_and_summary
User->>FastAPIRoute: POST /activate_conversation
FastAPIRoute->>Runtime: get_session(channel_id)
Runtime-->>FastAPIRoute: runtime
FastAPIRoute->>Store: get_conversation_window(conv_id, MAX_CONVERSATION_TURNS_IN_MEMORY)
Store-->>FastAPIRoute: conv (turns window)
FastAPIRoute->>Utils: restore_history_from_turns(conv["turns"])
Utils-->>FastAPIRoute: restored_history
FastAPIRoute->>Runtime: set execution_context._conversation_history, durable_turn_count
FastAPIRoute->>LabelHelper: try_ensure_topic_and_summary(runtime, generate_topic_and_summary)
LabelHelper->>StoreHelper: ensure_topic_and_summary(runtime, generate_topic_and_summary, turns_appended=0)
StoreHelper->>Store: get_conversation_label_state(conversation_id)
Store-->>StoreHelper: (stored_topic, durable_turns)
StoreHelper->>StoreHelper: _label_is_due(stored_topic, durable_turns, 0)
alt blank topic and durable_turns > 0
StoreHelper->>Store: get_conversation_summaries(conversation_id)
Store-->>StoreHelper: turns
StoreHelper->>LLM: generate_topic_and_summary(turns)
LLM-->>StoreHelper: (topic, summary)
StoreHelper->>Store: update_conversation_topic_summary(conversation_id, topic, summary)
else already labeled or no turns
StoreHelper-->>LabelHelper: None
end
LabelHelper-->>FastAPIRoute: (topic, summary) or None
FastAPIRoute-->>User: {status: "ok"}
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
test_invoke_command_wraps_error had been skipping on a hasattr() guard for CommandExecutor._invoke_command_metadata_extraction_workflow, so it asserted nothing instead of failing when the implementation moved. Two things had gone stale, not one: the CME hop is now a perform_action call with command_name "wildcard", and ChatSession no longer takes workflow_folderpath/workflow_id_str, so the session the test built could not have been constructed either. Rewritten against the production caller. workflow_execution_context.py:1175 passes `self` to invoke_command, so the test drives a real WorkflowExecutionContext rather than a ChatSession — the parameter is annotated ChatSession but only cme_workflow, get_active_workflow() and cme_workflow._context are touched, and WEC supplies all three. The active workflow is pushed as production pushes it, because a WEC's get_active_workflow() reads only the context-local stack and deliberately does not fall back to the bound app workflow. perform_action is the stub seam because that IS the CME hop now; the stubbed registry cannot serve it (DummyCRD has no class for "wildcard", so perform_action would raise ValueError over the RuntimeError under test). Only the CME hop routes through perform_action — invoke_command instantiates the resolved command's generator directly — so stubbing it does not hide the call being tested. Verified discriminating rather than assumed: with a non-raising generator in the same wiring, invoke_command returns normally with the response text and command_name set, so the RuntimeError is attributable to the faulty generator reached through invoke_command. Swap the generator and the test fails with DID NOT RAISE. This was the only hasattr-guarded skip of its kind in the suite. Co-authored-by: Cursor <cursoragent@cursor.com>
Six verified-wrong claims in the agent-facing docs, found while releasing v3.1.0. No version bump: documentation only. - Suite size and duration. AGENTS.md said ~24 minutes and the skills said 495 collected; it is 1803 tests and 48:56, measured 2026-08-13 at v3.1.0. An agent budgeting against ~24 minutes concludes a healthy run has hung. Also updated in two helper scripts whose sanity thresholds keyed off 495. - The full-run baseline the validation skill asked for. Its row said "unverified — run it and record it"; recorded as 1788 passed / 15 skipped / 0 failed / 0 errors, with the honest caveat that this was ONE run and so does not meet that skill's own two-consecutive-runs idempotence bar, plus the checksum finding that 10 regenerable build-cache JSONs are rewritten by the build tests. - The 0.5s autouse sleep. Documented as costing ~4 min of dead time with the underlying thread-lifecycle issue unfixed; conftest.py:151-169 has since been changed to join only when a ChatWorker is actually alive. - `bd close --reason`. Held across THIRTEEN skill files as silently failing to persist. Retested on bd 1.1.2: twelve consecutive closes with --reason all landed, verified by reading close_reason back out of the JSONL. Corrected in every one of those files rather than only the owning skill — a library that contradicts itself is worse than one uniformly stale. The 2026-06-11 observation is kept, attributed to the older bd, because the symptom may return if Dolt wedges; the verify-the-JSONL half of that rule is left as permanent. - The publication boundary. change-control section 7 said `.claude/skills/` was untracked-but-not-gitignored, making `git add -A` a publication risk. It is now a mix: 14 skill dirs plus README are deliberately tracked and public, while the four team-private ones are gitignored by exact path (.gitignore:219-237), as are the tau2 plan, RSI harness report and Forge spec. So -A can no longer publish them. Kept the stage-by-explicit-path habit, with its real remaining rationale. - Recorded what a stale-old_value chain in .beads/interactions.jsonl means, since an audit found 225 of them: it is the flakiness fingerprint, it is harmless (old_value is a breadcrumb nothing reads; every status was correct), and that log must not be "repaired" because it is the only evidence those episodes left. The four gitignored team-private skills carry the same --reason correction locally; they are excluded from this commit by .gitignore, as intended. Co-authored-by: Cursor <cursoragent@cursor.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #65, plus ten further findings from an adversarial review of it. Full findings and resolutions are in bd epic
fix-dzs(11 children, all closed).Summary
Process shutdown no longer calls
LLM_CONVERSATION_STORE. Conversations are labeled when they are actually used as a chat instead.The harm wasn't a missing title.
finalize_conversations_on_shutdownran up tomax_live_sessions(50) sequential, untimed LLM calls, and it ran beforestop_all_chat_sessionswrites checkpoints — so under a 30s termination grace period the process was killed generating titles and lost everything each channel had accumulated since its last eviction. Shutdown labeling was also already LRU roulette: in the soak that motivated #65, the ~92% of conversations evicted before SIGTERM were never labeled at all.save_conversation_incrementalchokepoint the turn path uses, and leaves blank placeholders. Theactive_conversation_id == 0fallback is kept deliberately — it is the only path that records turns for a channel whose persist was skipped by a raisingwork_fn.DONE, so the call is outside the drain, off the turn's critical path, and outside the 409 window), on activate when still blank, and on rotate. Refreshed at turn counts 1, 4, 16, 64, … so a long thread costs O(log n) calls; labeling once and never again would give a 40-turn thread the title of its first exchange.timeoutandnum_retries=1go todspy.LM.asyncio.wait_forcancels the wait while the thread runs on, and CPython then joins default-executor threads for up toTHREAD_JOIN_TIMEOUT(300s) at exit. NewLLM_CONVERSATION_STORE_TIMEOUT_SECONDS(default 12), resolved process-environment-first so a container override is reachable, validated and logged at startup."conversation summary"now identifies its turn on the deterministic and action paths. Both hardcoded a constant ("assistant_mode_command","process_action command"), and sincegenerate_topic_and_summaryreads only that field,/initializeconversations were asking an LLM to title a fixed string.' 1', counting as labeled forever._ensure_unique_topicexcludes the record being written. A rewrite with an unchanged topic oscillated between'T'and'T 1'.get_conversation_window. The restore paths kept 20 turns but read every turn, so at 450 KB payloads restoring a long conversation was tens of MB resident./new_conversationpersists before labeling. In the reserved-but-unwritten state it spent a generation and then raised, making the rotate that would have cleaned the state up the one unavailable operation.on_doneonrun_owned_turn).invoke_agent, the MCP-exposed tool, is the streaming endpoint, and fastapi-mcp reads the whole stream as the tool result.Three of #65's specific designs were wrong
Corrected here rather than implemented as written:
previewrested on"conversation summary"being meaningful; it was a constant literal on exactly the paths the motivating workload uses. Deferred and closed with the research recorded — a blank topic turns out to be harmless anyway (get_conversation_by_topichas no production caller; activate takes aconversation_id).THREAD_JOIN_TIMEOUTreason above.runtime.lock") would have put the LLM call back inside the 30s drain it was written to protect, added its latency to the user's response, and widened the 409 window.Test plan
.pthorthreshold.jsontouched (thefix-0hbclass of incident did not recur)ValueError), and streaming EOFrun_owned_turnlabeling now covered at all (it had no test; only_run_turn's did)Reviewer notes
docs/gets no new design doc: this was a review of an existing public issue rather than a new design, so the bd epic and #65 are the design record (agreed with the maintainer). Two pre-existing Sourcery style warnings inconversation_store.pyandutils.pyare untouched. The suite is 1803 tests, not the ~495 our docs claim — that doc rot is not fixed here.Made with Cursor
Summary by Sourcery
Implement lazy, client-bounded conversation labeling and remove LLM usage from shutdown, improving reliability, performance, and operational control for FastWorkflow conversation management.
New Features:
Bug Fixes:
Enhancements:
Documentation:
Tests: