fix(simulation): prevent concurrent prepare threads - #64
Open
MA1503 wants to merge 1 commit into
Open
Conversation
The prepare status endpoint reported "not_started" while a prepare was actively running (it only checks output files, not the live task), and /prepare had no duplicate guard — every visit to Step 2 spawned another full persona-generation thread. Observed: three concurrent generators (counter bands 30/139/159 of 252) splitting Ollama throughput three ways and racing on the same profile files, with jumping counters in the UI. Fix: a module-level registry of active prepare tasks per simulation. /prepare refuses duplicates and returns the existing task_id; /prepare/status reports active tasks as "preparing" with real progress. The registry entry is cleared when the thread ends (success and failure). After a backend restart the registry is empty, so a stale "preparing" state does not block a fresh prepare. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
Opening Step 2 (environment setup) in the frontend while a preparation is
already running spawns another full persona-generation thread. With repeated
visits, several generators run at once — they split the Ollama throughput
between them and race on the same profile output files. The UI shows
progress counters that jump around (observed: three concurrent generators at
counter bands 30 / 139 / 159 of 252).
Cause
Two gaps in
backend/app/api/simulation.py:/prepare/statusreportsnot_startedwhile a prepare is running.The status endpoint only checks whether the output files already exist
(
_check_simulation_prepared). While generation is still in flight thosefiles are absent, so with a
simulation_idand notask_idthe endpointreturns
"status": "not_started"— telling the frontend that nopreparation has begun.
/preparehas no duplicate guard. It creates a task and immediatelystarts
threading.Thread(target=run_prepare)with no record of anin-flight prepare, so nothing stops a second (or third) call from starting
another thread for the same simulation.
Combined, a client that polls
/prepare/status, seesnot_started, and calls/prepareagain keeps launching new generation threads.Fix
A module-level registry
_active_preparesmapssimulation_id → task_id:/preparerefuses to start a second prepare while one is active for thesame simulation and returns the existing
task_id(statuspreparing)instead of spawning another thread. A stale/finished entry is cleared so a
genuine re-prepare still works.
finallyblock — on success and on failure./prepare/statusnow checks the registry: when a task isPENDINGorPROCESSINGit reportspreparingwith the task's real progress andmessage, instead of the misleading
not_started.After a backend restart the registry is empty, so a stale
preparingvalue instate.jsoncannot block a fresh prepare.Repro
(or poll
/prepare/statuswith onlysimulation_id, seenot_started,and call
/prepareagain).counters jump, profile files are overwritten by racing writers.
After the fix: subsequent
/preparecalls return the existing task and nosecond thread starts;
/prepare/statusreportspreparingwith realprogress throughout.
🤖 Generated with Claude Code