fix: resilience fixes for local LLM structured output#152
Closed
82deutschmark wants to merge 5 commits intoPlanExeOrg:mainfrom
Closed
fix: resilience fixes for local LLM structured output#15282deutschmark wants to merge 5 commits intoPlanExeOrg:mainfrom
82deutschmark wants to merge 5 commits intoPlanExeOrg:mainfrom
Conversation
- select_scenario.py: make holistic_profile_of_the_plan optional with default='' to handle LLMs that truncate output before last field in nested JSON objects (confirmed fix: SelectScenarioTask now passes with Qwen 3.5-35B-A3B at num_output=8192) - CreateScheduleTask: fix KeyError on server_iso_utc for CLI runs Tested on Mac Mini M4 Pro 64GB with LM Studio + Qwen 3.5-35B-A3B. Root cause: LMStudio default num_output=4096 causes silent truncation of last fields in structured JSON output.
Collaborator
Author
|
Closing — PR contains local config files and run outputs that should not be in upstream. Reopening with only the relevant code fixes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two bug fixes discovered during local model QA testing on Mac Mini M4 Pro 64GB with Qwen 3.5-35B-A3B via LM Studio.
Fix 1: SelectScenarioTask — holistic_profile_of_the_plan field
Problem:
SelectScenarioTaskfails with Pydantic validation error onholistic_profile_of_the_planfield inPlanCharacteristics. Field is defined last in the schema; local LLMs with constrained output budgets close the JSON object before generating it.Confirmed root cause: LMStudio default
num_output=4096causes silent truncation. Reproduced deterministically with Qwen 3.5-35B-A3B (2 consecutive failures, 0 retries succeed).Fix:
default=''makes the field resilient — pipeline continues with an empty holistic profile rather than failing entirely. Downstreamscenarios_markdown.py(line 71) uses f-string interpolation so empty string renders cleanly.Verified fix: With
num_output=8192,SelectScenarioTaskpasses.default=''adds a second layer of resilience for other local models.Fix 2: CreateScheduleTask — KeyError on server_iso_utc
Problem:
CreateScheduleTaskraisesKeyError: 'server_iso_utc'during CLI runs.Fix: Safe dict access with
.get().Test environment